Attaching to ozonesecure_scm_1, ozonesecure_datanode_3, ozonesecure_s3g_1, ozonesecure_kms_1, ozonesecure_datanode_2, ozonesecure_datanode_1, ozonesecure_kdc_1, ozonesecure_recon_1, ozonesecure_om_1 scm_1 | Sleeping for 5 seconds scm_1 | Setting up kerberos!! scm_1 | KDC ISSUER_SERVER => kdc:8081 scm_1 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found datanode_3 | Sleeping for 5 seconds datanode_3 | Setting up kerberos!! scm_1 | Sleeping for seconds datanode_3 | KDC ISSUER_SERVER => kdc:8081 scm_1 | Got 200, KDC service ready!! datanode_3 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found scm_1 | Download dn/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab datanode_3 | Sleeping for seconds scm_1 | --2019-10-07 08:08:20-- http://kdc:8081/keytab/scm/dn datanode_3 | Got 200, KDC service ready!! datanode_3 | Download dn/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab datanode_3 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/d8b9c0e14ce8/dn datanode_3 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_3 | HTTP request sent, awaiting response... 200 OK kms_1 | Sleeping for 5 seconds scm_1 | HTTP request sent, awaiting response... 200 OK datanode_3 | Length: 158 [application/octet-stream] s3g_1 | Sleeping for 5 seconds om_1 | Sleeping for 5 seconds kdc_1 | Issuer is listening on : 8081krb5kdc: starting... kms_1 | Setting up kerberos!! scm_1 | Length: 140 [application/octet-stream] datanode_3 | Saving to: '/etc/security/keytabs/dn.keytab' s3g_1 | Setting up kerberos!! datanode_1 | Sleeping for 5 seconds om_1 | Setting up kerberos!! datanode_2 | Sleeping for 5 seconds kdc_1 | kadmind: starting... kms_1 | KDC ISSUER_SERVER => kdc:8081 scm_1 | Saving to: '/etc/security/keytabs/dn.keytab' datanode_3 | s3g_1 | KDC ISSUER_SERVER => kdc:8081 datanode_1 | Setting up kerberos!! om_1 | KDC ISSUER_SERVER => kdc:8081 datanode_2 | Setting up kerberos!! kdc_1 | otp: Loaded kms_1 | /opt/starter.sh: line 66: SLEEP_SECONDS: command not found scm_1 | s3g_1 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found datanode_3 | 0K 100% 12.5M=0s datanode_1 | KDC ISSUER_SERVER => kdc:8081 recon_1 | Sleeping for 5 seconds om_1 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found datanode_2 | KDC ISSUER_SERVER => kdc:8081 kdc_1 | Oct 07 08:08:00 kdc krb5kdc[9](info): setting up network... kms_1 | Sleeping for seconds scm_1 | 0K 100% 11.4M=0s s3g_1 | Sleeping for seconds datanode_3 | datanode_1 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found recon_1 | Waiting for the service om:9874 om_1 | Sleeping for seconds datanode_2 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found kdc_1 | krb5kdc: setsockopt(9,IPV6_V6ONLY,1) worked kms_1 | Got 200, KDC service ready!! scm_1 | s3g_1 | Got 200, KDC service ready!! datanode_3 | 2019-10-07 08:08:21 (12.5 MB/s) - '/etc/security/keytabs/dn.keytab' saved [158/158] datanode_1 | Sleeping for seconds recon_1 | Setting up kerberos!! om_1 | Got 200, KDC service ready!! datanode_2 | Sleeping for seconds kdc_1 | krb5kdc: setsockopt(11,IPV6_V6ONLY,1) worked kms_1 | Download dn/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab scm_1 | 2019-10-07 08:08:21 (11.4 MB/s) - '/etc/security/keytabs/dn.keytab' saved [140/140] s3g_1 | Download dn/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab datanode_3 | datanode_1 | Got 200, KDC service ready!! recon_1 | KDC ISSUER_SERVER => kdc:8081 datanode_2 | Got 200, KDC service ready!! om_1 | Download dn/om@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab kdc_1 | Oct 07 08:08:00 kdc krb5kdc[9](info): set up 4 sockets kms_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/c8a6426e556a/dn scm_1 | s3g_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/s3g/dn datanode_3 | Keytab name: FILE:/etc/security/keytabs/dn.keytab datanode_1 | Download dn/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab recon_1 | /opt/hadoop/bin/docker/entrypoint.sh: line 66: SLEEP_SECONDS: command not found datanode_2 | Download dn/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab om_1 | --2019-10-07 08:08:20-- http://kdc:8081/keytab/om/dn kdc_1 | Oct 07 08:08:00 kdc krb5kdc[9](info): commencing operation kms_1 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | Keytab name: FILE:/etc/security/keytabs/dn.keytab s3g_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | KVNO Timestamp Principal datanode_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/ad18a6fa7980/dn recon_1 | Sleeping for seconds datanode_2 | --2019-10-07 08:08:20-- http://kdc:8081/keytab/311de0fd64d4/dn om_1 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Oct 07 08:07:50 57f879621cdb kadmin.local[1](info): No dictionary file specified, continuing without one. kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | KVNO Timestamp Principal s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_3 | ---- ----------------- -------------------------------------------------------- datanode_1 | Resolving kdc (kdc)... 172.18.0.4 recon_1 | Got 200, KDC service ready!! datanode_2 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Oct 07 08:07:52 8a3377033a2b kadmin.local[1](info): No dictionary file specified, continuing without one. kms_1 | HTTP request sent, awaiting response... 200 OK scm_1 | ---- ----------------- -------------------------------------------------------- om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | HTTP request sent, awaiting response... 200 OK datanode_3 | 2 10/07/19 08:08:21 dn/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. recon_1 | Download dn/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/dn.keytab datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | Oct 07 08:08:04 kdc kadmind[15](info): No dictionary file specified, continuing without one. kms_1 | Length: 158 [application/octet-stream] scm_1 | 2 10/07/19 08:08:20 dn/scm@EXAMPLE.COM om_1 | HTTP request sent, awaiting response... 200 OK s3g_1 | Length: 140 [application/octet-stream] datanode_1 | HTTP request sent, awaiting response... 200 OK datanode_3 | 2 10/07/19 08:08:21 dn/d8b9c0e14ce8@EXAMPLE.COM recon_1 | --2019-10-07 08:08:42-- http://kdc:8081/keytab/recon/dn datanode_2 | HTTP request sent, awaiting response... 200 OK kdc_1 | Oct 07 08:08:04 kdc kadmind[15](info): setting up network... kms_1 | Saving to: '/etc/security/keytabs/dn.keytab' scm_1 | 2 10/07/19 08:08:21 dn/scm@EXAMPLE.COM om_1 | Length: 138 [application/octet-stream] s3g_1 | Saving to: '/etc/security/keytabs/dn.keytab' datanode_1 | Length: 158 [application/octet-stream] datanode_3 | Download om/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab recon_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | Length: 158 [application/octet-stream] kdc_1 | kadmind: setsockopt(9,IPV6_V6ONLY,1) worked kms_1 | scm_1 | Download om/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab om_1 | Saving to: '/etc/security/keytabs/dn.keytab' s3g_1 | datanode_1 | Saving to: '/etc/security/keytabs/dn.keytab' datanode_3 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/d8b9c0e14ce8/om recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | Saving to: '/etc/security/keytabs/dn.keytab' kdc_1 | kadmind: setsockopt(11,IPV6_V6ONLY,1) worked kms_1 | 0K 100% 8.00M=0s scm_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/scm/om om_1 | s3g_1 | 0K 100% 8.44M=0s datanode_3 | Resolving kdc (kdc)... 172.18.0.4 datanode_1 | recon_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | kdc_1 | kadmind: setsockopt(13,IPV6_V6ONLY,1) worked kms_1 | scm_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | 0K 100% 10.2M=0s s3g_1 | datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. recon_1 | Length: 144 [application/octet-stream] datanode_1 | 0K 100% 8.16M=0s datanode_2 | 0K 100% 9.03M=0s kdc_1 | Oct 07 08:08:04 kdc kadmind[15](info): set up 6 sockets kms_1 | 2019-10-07 08:08:21 (8.00 MB/s) - '/etc/security/keytabs/dn.keytab' saved [158/158] scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | s3g_1 | 2019-10-07 08:08:22 (8.44 MB/s) - '/etc/security/keytabs/dn.keytab' saved [140/140] datanode_3 | HTTP request sent, awaiting response... 200 OK recon_1 | Saving to: '/etc/security/keytabs/dn.keytab' datanode_1 | datanode_2 | kdc_1 | Oct 07 08:08:04 kdc kadmind[15](info): Seeding random number generator kms_1 | scm_1 | HTTP request sent, awaiting response... 200 OK om_1 | 2019-10-07 08:08:20 (10.2 MB/s) - '/etc/security/keytabs/dn.keytab' saved [138/138] s3g_1 | datanode_3 | Length: 158 [application/octet-stream] recon_1 | datanode_1 | 2019-10-07 08:08:21 (8.16 MB/s) - '/etc/security/keytabs/dn.keytab' saved [158/158] datanode_2 | 2019-10-07 08:08:20 (9.03 MB/s) - '/etc/security/keytabs/dn.keytab' saved [158/158] kdc_1 | Oct 07 08:08:04 kdc kadmind[15](info): starting scm_1 | Length: 140 [application/octet-stream] kms_1 | Keytab name: FILE:/etc/security/keytabs/dn.keytab om_1 | s3g_1 | Keytab name: FILE:/etc/security/keytabs/dn.keytab datanode_3 | Saving to: '/etc/security/keytabs/om.keytab' recon_1 | 0K 100% 7.60M=0s datanode_1 | datanode_2 | kdc_1 | Generiting keytab scm_1 | Saving to: '/etc/security/keytabs/om.keytab' kms_1 | KVNO Timestamp Principal om_1 | Keytab name: FILE:/etc/security/keytabs/dn.keytab s3g_1 | KVNO Timestamp Principal datanode_3 | recon_1 | datanode_1 | Keytab name: FILE:/etc/security/keytabs/dn.keytab datanode_2 | Keytab name: FILE:/etc/security/keytabs/dn.keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | kms_1 | ---- ----------------- -------------------------------------------------------- om_1 | KVNO Timestamp Principal s3g_1 | ---- ----------------- -------------------------------------------------------- datanode_3 | 0K 100% 20.3M=0s recon_1 | 2019-10-07 08:08:43 (7.60 MB/s) - '/etc/security/keytabs/dn.keytab' saved [144/144] datanode_1 | KVNO Timestamp Principal datanode_2 | KVNO Timestamp Principal kdc_1 | Oct 07 08:08:09 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 0K 100% 10.5M=0s kms_1 | 2 10/07/19 08:08:21 dn/c8a6426e556a@EXAMPLE.COM om_1 | ---- ----------------- -------------------------------------------------------- s3g_1 | 2 10/07/19 08:08:22 dn/s3g@EXAMPLE.COM datanode_3 | recon_1 | datanode_1 | ---- ----------------- -------------------------------------------------------- datanode_2 | ---- ----------------- -------------------------------------------------------- kdc_1 | Oct 07 08:08:09 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435689, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | kms_1 | 2 10/07/19 08:08:21 dn/c8a6426e556a@EXAMPLE.COM om_1 | 2 10/07/19 08:08:20 dn/om@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:22 dn/s3g@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:22 (20.3 MB/s) - '/etc/security/keytabs/om.keytab' saved [158/158] datanode_1 | 2 10/07/19 08:08:21 dn/ad18a6fa7980@EXAMPLE.COM recon_1 | Keytab name: FILE:/etc/security/keytabs/dn.keytab datanode_2 | 2 10/07/19 08:08:20 dn/311de0fd64d4@EXAMPLE.COM kdc_1 | WARNING: no policy specified for test/test@EXAMPLE.COM; defaulting to no policy scm_1 | 2019-10-07 08:08:22 (10.5 MB/s) - '/etc/security/keytabs/om.keytab' saved [140/140] kms_1 | Download om/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab om_1 | 2 10/07/19 08:08:20 dn/om@EXAMPLE.COM s3g_1 | Download om/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab datanode_3 | datanode_1 | 2 10/07/19 08:08:21 dn/ad18a6fa7980@EXAMPLE.COM recon_1 | KVNO Timestamp Principal kdc_1 | Principal "test/test@EXAMPLE.COM" created. scm_1 | kms_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/c8a6426e556a/om om_1 | Download om/om@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab s3g_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/s3g/om datanode_1 | Download om/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab datanode_3 | Keytab name: FILE:/etc/security/keytabs/om.keytab datanode_2 | 2 10/07/19 08:08:20 dn/311de0fd64d4@EXAMPLE.COM recon_1 | ---- ----------------- -------------------------------------------------------- kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | Keytab name: FILE:/etc/security/keytabs/om.keytab kms_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | --2019-10-07 08:08:20-- http://kdc:8081/keytab/om/om s3g_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/ad18a6fa7980/om datanode_3 | KVNO Timestamp Principal datanode_2 | Download om/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab recon_1 | 2 10/07/19 08:08:43 dn/recon@EXAMPLE.COM scm_1 | KVNO Timestamp Principal kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | ---- ----------------- -------------------------------------------------------- datanode_2 | --2019-10-07 08:08:20-- http://kdc:8081/keytab/311de0fd64d4/om recon_1 | 2 10/07/19 08:08:43 dn/recon@EXAMPLE.COM scm_1 | ---- ----------------- -------------------------------------------------------- kms_1 | HTTP request sent, awaiting response... 200 OK om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | HTTP request sent, awaiting response... 200 OK kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_3 | 2 10/07/19 08:08:22 om/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | Resolving kdc (kdc)... 172.18.0.4 recon_1 | Download om/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/om.keytab scm_1 | 2 10/07/19 08:08:22 om/scm@EXAMPLE.COM kms_1 | Length: 158 [application/octet-stream] om_1 | HTTP request sent, awaiting response... 200 OK s3g_1 | Length: 140 [application/octet-stream] kdc_1 | Entry for principal test/test@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/test.test.keytab. datanode_3 | 2 10/07/19 08:08:22 om/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. recon_1 | --2019-10-07 08:08:43-- http://kdc:8081/keytab/recon/om scm_1 | 2 10/07/19 08:08:22 om/scm@EXAMPLE.COM kms_1 | Saving to: '/etc/security/keytabs/om.keytab' om_1 | Length: 138 [application/octet-stream] s3g_1 | Saving to: '/etc/security/keytabs/om.keytab' datanode_3 | Download scm/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab kdc_1 | Entry for principal test/test@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/test.test.keytab. datanode_1 | Length: 158 [application/octet-stream] datanode_2 | HTTP request sent, awaiting response... 200 OK recon_1 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | Download scm/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab kms_1 | om_1 | Saving to: '/etc/security/keytabs/om.keytab' s3g_1 | datanode_3 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/d8b9c0e14ce8/scm kdc_1 | Generiting keytab datanode_1 | Saving to: '/etc/security/keytabs/om.keytab' datanode_2 | Length: 158 [application/octet-stream] recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/scm/scm om_1 | s3g_1 | 0K 100% 9.65M=0s kms_1 | 0K 100% 18.7M=0s datanode_3 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_1 | datanode_2 | Saving to: '/etc/security/keytabs/om.keytab' recon_1 | HTTP request sent, awaiting response... 200 OK scm_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | 0K 100% 10.2M=0s s3g_1 | kms_1 | datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | WARNING: no policy specified for test/test@EXAMPLE.COM; defaulting to no policy datanode_1 | 0K 100% 11.7M=0s datanode_2 | recon_1 | Length: 144 [application/octet-stream] scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | s3g_1 | 2019-10-07 08:08:22 (9.65 MB/s) - '/etc/security/keytabs/om.keytab' saved [140/140] kms_1 | 2019-10-07 08:08:22 (18.7 MB/s) - '/etc/security/keytabs/om.keytab' saved [158/158] datanode_3 | HTTP request sent, awaiting response... 200 OK kdc_1 | add_principal: Principal or policy already exists while creating "test/test@EXAMPLE.COM". datanode_1 | datanode_2 | 0K 100% 13.1M=0s recon_1 | Saving to: '/etc/security/keytabs/om.keytab' scm_1 | HTTP request sent, awaiting response... 200 OK om_1 | 2019-10-07 08:08:21 (10.2 MB/s) - '/etc/security/keytabs/om.keytab' saved [138/138] s3g_1 | kms_1 | datanode_3 | Length: 160 [application/octet-stream] kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_1 | 2019-10-07 08:08:22 (11.7 MB/s) - '/etc/security/keytabs/om.keytab' saved [158/158] datanode_2 | scm_1 | Length: 142 [application/octet-stream] recon_1 | om_1 | s3g_1 | Keytab name: FILE:/etc/security/keytabs/om.keytab kms_1 | Keytab name: FILE:/etc/security/keytabs/om.keytab datanode_3 | Saving to: '/etc/security/keytabs/scm.keytab' kdc_1 | Entry for principal test/test@EXAMPLE.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/test.test.keytab. datanode_1 | datanode_2 | 2019-10-07 08:08:21 (13.1 MB/s) - '/etc/security/keytabs/om.keytab' saved [158/158] scm_1 | Saving to: '/etc/security/keytabs/scm.keytab' recon_1 | 0K 100% 16.4M=0s om_1 | Keytab name: FILE:/etc/security/keytabs/om.keytab s3g_1 | KVNO Timestamp Principal kms_1 | KVNO Timestamp Principal datanode_3 | kdc_1 | Entry for principal test/test@EXAMPLE.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/test.test.keytab. datanode_1 | Keytab name: FILE:/etc/security/keytabs/om.keytab datanode_2 | scm_1 | recon_1 | om_1 | KVNO Timestamp Principal s3g_1 | ---- ----------------- -------------------------------------------------------- kms_1 | ---- ----------------- -------------------------------------------------------- datanode_3 | 0K 100% 12.7M=0s kdc_1 | Generiting keytab datanode_1 | KVNO Timestamp Principal datanode_2 | Keytab name: FILE:/etc/security/keytabs/om.keytab scm_1 | 0K 100% 11.8M=0s recon_1 | 2019-10-07 08:08:43 (16.4 MB/s) - '/etc/security/keytabs/om.keytab' saved [144/144] om_1 | ---- ----------------- -------------------------------------------------------- s3g_1 | 2 10/07/19 08:08:22 om/s3g@EXAMPLE.COM kms_1 | 2 10/07/19 08:08:22 om/c8a6426e556a@EXAMPLE.COM datanode_3 | kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_1 | ---- ----------------- -------------------------------------------------------- datanode_2 | KVNO Timestamp Principal scm_1 | om_1 | 2 10/07/19 08:08:21 om/om@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:22 om/s3g@EXAMPLE.COM kms_1 | 2 10/07/19 08:08:22 om/c8a6426e556a@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:23 (12.7 MB/s) - '/etc/security/keytabs/scm.keytab' saved [160/160] recon_1 | kdc_1 | WARNING: no policy specified for test/test@EXAMPLE.COM; defaulting to no policy datanode_1 | 2 10/07/19 08:08:22 om/ad18a6fa7980@EXAMPLE.COM datanode_2 | ---- ----------------- -------------------------------------------------------- scm_1 | 2019-10-07 08:08:22 (11.8 MB/s) - '/etc/security/keytabs/scm.keytab' saved [142/142] s3g_1 | Download scm/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab om_1 | 2 10/07/19 08:08:21 om/om@EXAMPLE.COM kms_1 | Download scm/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab datanode_3 | kdc_1 | add_principal: Principal or policy already exists while creating "test/test@EXAMPLE.COM". recon_1 | Keytab name: FILE:/etc/security/keytabs/om.keytab datanode_1 | 2 10/07/19 08:08:22 om/ad18a6fa7980@EXAMPLE.COM datanode_2 | 2 10/07/19 08:08:21 om/311de0fd64d4@EXAMPLE.COM scm_1 | s3g_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/s3g/scm om_1 | Download scm/om@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab kms_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/c8a6426e556a/scm datanode_3 | Keytab name: FILE:/etc/security/keytabs/scm.keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. recon_1 | KVNO Timestamp Principal datanode_1 | Download scm/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab datanode_2 | 2 10/07/19 08:08:21 om/311de0fd64d4@EXAMPLE.COM scm_1 | Keytab name: FILE:/etc/security/keytabs/scm.keytab s3g_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/om/scm kms_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | KVNO Timestamp Principal kdc_1 | Entry for principal test/test@EXAMPLE.COM with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/test.test.keytab. recon_1 | ---- ----------------- -------------------------------------------------------- datanode_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/ad18a6fa7980/scm scm_1 | KVNO Timestamp Principal om_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | Download scm/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_3 | ---- ----------------- -------------------------------------------------------- kdc_1 | Entry for principal test/test@EXAMPLE.COM with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/test.test.keytab. recon_1 | 2 10/07/19 08:08:43 om/recon@EXAMPLE.COM datanode_1 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | ---- ----------------- -------------------------------------------------------- om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | --2019-10-07 08:08:21-- http://kdc:8081/keytab/311de0fd64d4/scm kms_1 | HTTP request sent, awaiting response... 200 OK kdc_1 | Generiting keytab datanode_3 | 2 10/07/19 08:08:23 scm/d8b9c0e14ce8@EXAMPLE.COM recon_1 | 2 10/07/19 08:08:43 om/recon@EXAMPLE.COM datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | HTTP request sent, awaiting response... 200 OK datanode_1 | HTTP request sent, awaiting response... 200 OK om_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. kms_1 | Length: 160 [application/octet-stream] datanode_3 | 2 10/07/19 08:08:23 scm/d8b9c0e14ce8@EXAMPLE.COM recon_1 | Download scm/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/scm.keytab scm_1 | 2 10/07/19 08:08:22 scm/scm@EXAMPLE.COM s3g_1 | Length: 142 [application/octet-stream] datanode_1 | Length: 160 [application/octet-stream] om_1 | Length: 140 [application/octet-stream] datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | WARNING: no policy specified for dn/om@EXAMPLE.COM; defaulting to no policy kms_1 | Saving to: '/etc/security/keytabs/scm.keytab' datanode_3 | Download HTTP/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab recon_1 | --2019-10-07 08:08:43-- http://kdc:8081/keytab/recon/scm scm_1 | 2 10/07/19 08:08:22 scm/scm@EXAMPLE.COM s3g_1 | Saving to: '/etc/security/keytabs/scm.keytab' datanode_1 | Saving to: '/etc/security/keytabs/scm.keytab' om_1 | Saving to: '/etc/security/keytabs/scm.keytab' datanode_2 | HTTP request sent, awaiting response... 200 OK kdc_1 | Principal "dn/om@EXAMPLE.COM" created. kms_1 | datanode_3 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/d8b9c0e14ce8/HTTP recon_1 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | Download HTTP/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab s3g_1 | datanode_1 | om_1 | datanode_2 | Length: 160 [application/octet-stream] kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. kms_1 | 0K 100% 12.0M=0s datanode_3 | Resolving kdc (kdc)... 172.18.0.4 recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/scm/HTTP s3g_1 | 0K 100% 10.6M=0s datanode_1 | 0K 100% 13.0M=0s om_1 | 0K 100% 16.5M=0s datanode_2 | Saving to: '/etc/security/keytabs/scm.keytab' kdc_1 | Entry for principal dn/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.om.keytab. datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kms_1 | recon_1 | HTTP request sent, awaiting response... 200 OK scm_1 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | datanode_1 | om_1 | datanode_2 | kdc_1 | Entry for principal dn/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.om.keytab. datanode_3 | HTTP request sent, awaiting response... 200 OK kms_1 | 2019-10-07 08:08:23 (12.0 MB/s) - '/etc/security/keytabs/scm.keytab' saved [160/160] recon_1 | Length: 146 [application/octet-stream] scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | 2019-10-07 08:08:23 (10.6 MB/s) - '/etc/security/keytabs/scm.keytab' saved [142/142] datanode_1 | 2019-10-07 08:08:23 (13.0 MB/s) - '/etc/security/keytabs/scm.keytab' saved [160/160] om_1 | 2019-10-07 08:08:22 (16.5 MB/s) - '/etc/security/keytabs/scm.keytab' saved [140/140] datanode_2 | 0K 100% 11.6M=0s kdc_1 | Generiting keytab datanode_3 | Length: 162 [application/octet-stream] kms_1 | recon_1 | Saving to: '/etc/security/keytabs/scm.keytab' scm_1 | HTTP request sent, awaiting response... 200 OK s3g_1 | datanode_1 | om_1 | datanode_2 | kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. kms_1 | Keytab name: FILE:/etc/security/keytabs/scm.keytab datanode_3 | Saving to: '/etc/security/keytabs/HTTP.keytab' recon_1 | scm_1 | Length: 144 [application/octet-stream] s3g_1 | Keytab name: FILE:/etc/security/keytabs/scm.keytab datanode_1 | Keytab name: FILE:/etc/security/keytabs/scm.keytab om_1 | Keytab name: FILE:/etc/security/keytabs/scm.keytab datanode_2 | 2019-10-07 08:08:22 (11.6 MB/s) - '/etc/security/keytabs/scm.keytab' saved [160/160] kdc_1 | WARNING: no policy specified for dn/311de0fd64d4@EXAMPLE.COM; defaulting to no policy kms_1 | KVNO Timestamp Principal datanode_3 | recon_1 | 0K 100% 15.5M=0s scm_1 | Saving to: '/etc/security/keytabs/HTTP.keytab' s3g_1 | KVNO Timestamp Principal datanode_1 | KVNO Timestamp Principal om_1 | KVNO Timestamp Principal datanode_2 | kdc_1 | Principal "dn/311de0fd64d4@EXAMPLE.COM" created. kms_1 | ---- ----------------- -------------------------------------------------------- datanode_3 | 0K 100% 13.0M=0s recon_1 | scm_1 | s3g_1 | ---- ----------------- -------------------------------------------------------- datanode_1 | ---- ----------------- -------------------------------------------------------- om_1 | ---- ----------------- -------------------------------------------------------- datanode_2 | Keytab name: FILE:/etc/security/keytabs/scm.keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. kms_1 | 2 10/07/19 08:08:23 scm/c8a6426e556a@EXAMPLE.COM datanode_3 | recon_1 | 2019-10-07 08:08:43 (15.5 MB/s) - '/etc/security/keytabs/scm.keytab' saved [146/146] scm_1 | 0K 100% 11.0M=0s datanode_1 | 2 10/07/19 08:08:23 scm/ad18a6fa7980@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:23 scm/s3g@EXAMPLE.COM datanode_2 | KVNO Timestamp Principal kdc_1 | Entry for principal dn/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.311de0fd64d4.keytab. kms_1 | 2 10/07/19 08:08:23 scm/c8a6426e556a@EXAMPLE.COM om_1 | 2 10/07/19 08:08:22 scm/om@EXAMPLE.COM recon_1 | scm_1 | datanode_3 | 2019-10-07 08:08:23 (13.0 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [162/162] datanode_1 | 2 10/07/19 08:08:23 scm/ad18a6fa7980@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:23 scm/s3g@EXAMPLE.COM datanode_2 | ---- ----------------- -------------------------------------------------------- kms_1 | Download HTTP/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab kdc_1 | Entry for principal dn/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.311de0fd64d4.keytab. om_1 | 2 10/07/19 08:08:22 scm/om@EXAMPLE.COM recon_1 | Keytab name: FILE:/etc/security/keytabs/scm.keytab scm_1 | 2019-10-07 08:08:23 (11.0 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [144/144] datanode_3 | datanode_1 | Download HTTP/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab s3g_1 | Download HTTP/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab datanode_2 | 2 10/07/19 08:08:22 scm/311de0fd64d4@EXAMPLE.COM kms_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/c8a6426e556a/HTTP kdc_1 | Generiting keytab om_1 | Download HTTP/om@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab recon_1 | KVNO Timestamp Principal scm_1 | datanode_3 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab datanode_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/ad18a6fa7980/HTTP s3g_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/s3g/HTTP datanode_2 | 2 10/07/19 08:08:22 scm/311de0fd64d4@EXAMPLE.COM kms_1 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. recon_1 | ---- ----------------- -------------------------------------------------------- om_1 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/om/HTTP scm_1 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab datanode_3 | KVNO Timestamp Principal datanode_1 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | Download HTTP/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | WARNING: no policy specified for dn/scm@EXAMPLE.COM; defaulting to no policy recon_1 | 2 10/07/19 08:08:43 scm/recon@EXAMPLE.COM om_1 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | KVNO Timestamp Principal datanode_3 | ---- ----------------- -------------------------------------------------------- datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | --2019-10-07 08:08:22-- http://kdc:8081/keytab/311de0fd64d4/HTTP kms_1 | HTTP request sent, awaiting response... 200 OK kdc_1 | Principal "dn/scm@EXAMPLE.COM" created. recon_1 | 2 10/07/19 08:08:43 scm/recon@EXAMPLE.COM om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | ---- ----------------- -------------------------------------------------------- datanode_3 | 2 10/07/19 08:08:23 HTTP/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | HTTP request sent, awaiting response... 200 OK kms_1 | Length: 162 [application/octet-stream] kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. recon_1 | Download HTTP/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/HTTP.keytab om_1 | HTTP request sent, awaiting response... 200 OK scm_1 | 2 10/07/19 08:08:23 HTTP/scm@EXAMPLE.COM datanode_3 | 2 10/07/19 08:08:23 HTTP/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | Length: 162 [application/octet-stream] datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | Length: 144 [application/octet-stream] kdc_1 | Oct 07 08:08:18 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 recon_1 | --2019-10-07 08:08:43-- http://kdc:8081/keytab/recon/HTTP om_1 | Length: 142 [application/octet-stream] scm_1 | 2 10/07/19 08:08:23 HTTP/scm@EXAMPLE.COM datanode_1 | Saving to: '/etc/security/keytabs/HTTP.keytab' datanode_3 | Download testuser/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab kms_1 | Saving to: '/etc/security/keytabs/HTTP.keytab' datanode_2 | HTTP request sent, awaiting response... 200 OK s3g_1 | Saving to: '/etc/security/keytabs/HTTP.keytab' kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 recon_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | Saving to: '/etc/security/keytabs/HTTP.keytab' datanode_1 | scm_1 | Download testuser/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab datanode_3 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/d8b9c0e14ce8/testuser kms_1 | datanode_2 | Length: 162 [application/octet-stream] s3g_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_create_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | datanode_1 | 0K 100% 12.6M=0s scm_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/scm/testuser datanode_3 | Resolving kdc (kdc)... 172.18.0.4 kms_1 | 0K 100% 13.5M=0s datanode_2 | Saving to: '/etc/security/keytabs/HTTP.keytab' s3g_1 | 0K 100% 10.4M=0s kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 recon_1 | HTTP request sent, awaiting response... 200 OK om_1 | 0K 100% 9.07M=0s datanode_1 | scm_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kms_1 | datanode_2 | s3g_1 | recon_1 | Length: 148 [application/octet-stream] kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_1 | 2019-10-07 08:08:23 (12.6 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [162/162] scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | kms_1 | 2019-10-07 08:08:24 (13.5 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [162/162] datanode_2 | 0K 100% 10.3M=0s datanode_3 | HTTP request sent, awaiting response... 200 OK s3g_1 | 2019-10-07 08:08:24 (10.4 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [144/144] recon_1 | Saving to: '/etc/security/keytabs/HTTP.keytab' kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | scm_1 | HTTP request sent, awaiting response... 200 OK om_1 | 2019-10-07 08:08:23 (9.07 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [142/142] kms_1 | datanode_3 | Length: 170 [application/octet-stream] s3g_1 | recon_1 | datanode_2 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab scm_1 | Length: 152 [application/octet-stream] om_1 | kms_1 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab datanode_3 | Saving to: '/etc/security/keytabs/testuser.keytab' s3g_1 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab datanode_2 | 2019-10-07 08:08:23 (10.3 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [162/162] recon_1 | 0K 100% 15.9M=0s datanode_1 | KVNO Timestamp Principal kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 scm_1 | Saving to: '/etc/security/keytabs/testuser.keytab' om_1 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab kms_1 | KVNO Timestamp Principal datanode_3 | datanode_2 | datanode_1 | ---- ----------------- -------------------------------------------------------- recon_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | om_1 | KVNO Timestamp Principal s3g_1 | KVNO Timestamp Principal kms_1 | ---- ----------------- -------------------------------------------------------- datanode_3 | 0K 100% 18.9M=0s datanode_2 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab datanode_1 | 2 10/07/19 08:08:23 HTTP/ad18a6fa7980@EXAMPLE.COM recon_1 | 2019-10-07 08:08:43 (15.9 MB/s) - '/etc/security/keytabs/HTTP.keytab' saved [148/148] kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 0K 100% 11.0M=0s om_1 | ---- ----------------- -------------------------------------------------------- s3g_1 | ---- ----------------- -------------------------------------------------------- kms_1 | 2 10/07/19 08:08:24 HTTP/c8a6426e556a@EXAMPLE.COM datanode_3 | datanode_2 | KVNO Timestamp Principal datanode_1 | 2 10/07/19 08:08:23 HTTP/ad18a6fa7980@EXAMPLE.COM kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_create_principal, test/test@EXAMPLE.COM, Principal or policy already exists, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | recon_1 | om_1 | 2 10/07/19 08:08:22 HTTP/om@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:24 HTTP/s3g@EXAMPLE.COM kms_1 | 2 10/07/19 08:08:24 HTTP/c8a6426e556a@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:24 (18.9 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [170/170] datanode_2 | ---- ----------------- -------------------------------------------------------- datanode_1 | Download testuser/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 recon_1 | Keytab name: FILE:/etc/security/keytabs/HTTP.keytab scm_1 | 2019-10-07 08:08:24 (11.0 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [152/152] om_1 | 2 10/07/19 08:08:23 HTTP/om@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:24 HTTP/s3g@EXAMPLE.COM datanode_2 | 2 10/07/19 08:08:23 HTTP/311de0fd64d4@EXAMPLE.COM datanode_3 | kms_1 | Download testuser/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab datanode_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/ad18a6fa7980/testuser kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 recon_1 | KVNO Timestamp Principal scm_1 | om_1 | Download testuser/om@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab s3g_1 | Download testuser/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab datanode_2 | 2 10/07/19 08:08:23 HTTP/311de0fd64d4@EXAMPLE.COM datanode_3 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab kms_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/c8a6426e556a/testuser datanode_1 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab recon_1 | ---- ----------------- -------------------------------------------------------- om_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/om/testuser s3g_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/s3g/testuser datanode_2 | Download testuser/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab datanode_3 | KVNO Timestamp Principal kms_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | KVNO Timestamp Principal recon_1 | 2 10/07/19 08:08:43 HTTP/recon@EXAMPLE.COM om_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/311de0fd64d4/testuser s3g_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | ---- ----------------- -------------------------------------------------------- kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_1 | HTTP request sent, awaiting response... 200 OK kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 scm_1 | ---- ----------------- -------------------------------------------------------- recon_1 | 2 10/07/19 08:08:43 HTTP/recon@EXAMPLE.COM om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_3 | 2 10/07/19 08:08:24 testuser/d8b9c0e14ce8@EXAMPLE.COM kms_1 | HTTP request sent, awaiting response... 200 OK datanode_1 | Length: 170 [application/octet-stream] kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2 10/07/19 08:08:24 testuser/scm@EXAMPLE.COM recon_1 | Download testuser/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser.keytab om_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | HTTP request sent, awaiting response... 200 OK kms_1 | Length: 170 [application/octet-stream] datanode_3 | 2 10/07/19 08:08:24 testuser/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | Saving to: '/etc/security/keytabs/testuser.keytab' kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2 10/07/19 08:08:24 testuser/scm@EXAMPLE.COM recon_1 | --2019-10-07 08:08:43-- http://kdc:8081/keytab/recon/testuser om_1 | Length: 150 [application/octet-stream] datanode_2 | HTTP request sent, awaiting response... 200 OK s3g_1 | Length: 152 [application/octet-stream] kms_1 | Saving to: '/etc/security/keytabs/testuser.keytab' datanode_3 | Download testuser2/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab datanode_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_create_principal, test/test@EXAMPLE.COM, Principal or policy already exists, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | Download testuser2/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab recon_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | Saving to: '/etc/security/keytabs/testuser.keytab' datanode_2 | Length: 170 [application/octet-stream] s3g_1 | Saving to: '/etc/security/keytabs/testuser.keytab' kms_1 | datanode_3 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/d8b9c0e14ce8/testuser2 kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 datanode_1 | 0K 100% 13.9M=0s scm_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/scm/testuser2 recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | datanode_2 | Saving to: '/etc/security/keytabs/testuser.keytab' s3g_1 | kms_1 | 0K 100% 12.9M=0s datanode_3 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_1 | scm_1 | Resolving kdc (kdc)... 172.18.0.4 recon_1 | HTTP request sent, awaiting response... 200 OK om_1 | 0K 100% 10.6M=0s datanode_2 | kms_1 | datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:24 (13.9 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [170/170] scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | 0K 100% 11.8M=0s recon_1 | Length: 156 [application/octet-stream] om_1 | datanode_2 | 0K 100% 12.6M=0s datanode_3 | HTTP request sent, awaiting response... 200 OK kms_1 | 2019-10-07 08:08:24 (12.9 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [170/170] scm_1 | HTTP request sent, awaiting response... 200 OK s3g_1 | recon_1 | Saving to: '/etc/security/keytabs/testuser.keytab' om_1 | 2019-10-07 08:08:23 (10.6 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [150/150] datanode_2 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_principal, test/test@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_3 | Length: 172 [application/octet-stream] datanode_1 | kms_1 | scm_1 | Length: 154 [application/octet-stream] s3g_1 | 2019-10-07 08:08:24 (11.8 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [152/152] recon_1 | om_1 | datanode_2 | 2019-10-07 08:08:23 (12.6 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [170/170] kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 datanode_3 | Saving to: '/etc/security/keytabs/testuser2.keytab' datanode_1 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab kms_1 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab scm_1 | Saving to: '/etc/security/keytabs/testuser2.keytab' s3g_1 | recon_1 | 0K 100% 18.4M=0s datanode_2 | om_1 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_3 | datanode_1 | KVNO Timestamp Principal kms_1 | KVNO Timestamp Principal scm_1 | s3g_1 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab recon_1 | datanode_2 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab om_1 | KVNO Timestamp Principal kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_3 | 0K 100% 19.6M=0s datanode_1 | ---- ----------------- -------------------------------------------------------- kms_1 | ---- ----------------- -------------------------------------------------------- scm_1 | 0K 100% 8.67M=0s s3g_1 | KVNO Timestamp Principal recon_1 | 2019-10-07 08:08:43 (18.4 MB/s) - '/etc/security/keytabs/testuser.keytab' saved [156/156] datanode_2 | KVNO Timestamp Principal om_1 | ---- ----------------- -------------------------------------------------------- kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_3 | kms_1 | 2 10/07/19 08:08:24 testuser/c8a6426e556a@EXAMPLE.COM s3g_1 | ---- ----------------- -------------------------------------------------------- scm_1 | datanode_1 | 2 10/07/19 08:08:24 testuser/ad18a6fa7980@EXAMPLE.COM recon_1 | datanode_2 | ---- ----------------- -------------------------------------------------------- kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 om_1 | 2 10/07/19 08:08:23 testuser/om@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:25 (19.6 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [172/172] kms_1 | 2 10/07/19 08:08:24 testuser/c8a6426e556a@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:24 testuser/s3g@EXAMPLE.COM scm_1 | 2019-10-07 08:08:25 (8.67 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [154/154] datanode_1 | 2 10/07/19 08:08:24 testuser/ad18a6fa7980@EXAMPLE.COM recon_1 | Keytab name: FILE:/etc/security/keytabs/testuser.keytab datanode_2 | 2 10/07/19 08:08:23 testuser/311de0fd64d4@EXAMPLE.COM kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2 10/07/19 08:08:23 testuser/om@EXAMPLE.COM datanode_3 | kms_1 | Download testuser2/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab s3g_1 | 2 10/07/19 08:08:24 testuser/s3g@EXAMPLE.COM scm_1 | datanode_1 | Download testuser2/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab recon_1 | KVNO Timestamp Principal datanode_2 | 2 10/07/19 08:08:23 testuser/311de0fd64d4@EXAMPLE.COM kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | Download testuser2/om@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab datanode_3 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab kms_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/c8a6426e556a/testuser2 s3g_1 | Download testuser2/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab scm_1 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab recon_1 | ---- ----------------- -------------------------------------------------------- datanode_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/ad18a6fa7980/testuser2 datanode_2 | Download testuser2/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/om/testuser2 kms_1 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/s3g/testuser2 scm_1 | KVNO Timestamp Principal datanode_3 | KVNO Timestamp Principal recon_1 | 2 10/07/19 08:08:43 testuser/recon@EXAMPLE.COM datanode_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | --2019-10-07 08:08:23-- http://kdc:8081/keytab/311de0fd64d4/testuser2 kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | ---- ----------------- -------------------------------------------------------- datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. recon_1 | 2 10/07/19 08:08:43 testuser/recon@EXAMPLE.COM om_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | Resolving kdc (kdc)... 172.18.0.4 kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | ---- ----------------- -------------------------------------------------------- s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_3 | 2 10/07/19 08:08:25 testuser2/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | HTTP request sent, awaiting response... 200 OK recon_1 | Download testuser2/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/testuser2.keytab om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 kms_1 | HTTP request sent, awaiting response... 200 OK scm_1 | 2 10/07/19 08:08:25 testuser2/scm@EXAMPLE.COM s3g_1 | HTTP request sent, awaiting response... 200 OK datanode_3 | 2 10/07/19 08:08:25 testuser2/d8b9c0e14ce8@EXAMPLE.COM datanode_1 | Length: 172 [application/octet-stream] recon_1 | --2019-10-07 08:08:43-- http://kdc:8081/keytab/recon/testuser2 om_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | HTTP request sent, awaiting response... 200 OK kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 kms_1 | Length: 172 [application/octet-stream] scm_1 | 2 10/07/19 08:08:25 testuser2/scm@EXAMPLE.COM s3g_1 | Length: 154 [application/octet-stream] datanode_3 | Download s3g/d8b9c0e14ce8@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab datanode_1 | Saving to: '/etc/security/keytabs/testuser2.keytab' recon_1 | Resolving kdc (kdc)... 172.18.0.4 om_1 | Length: 152 [application/octet-stream] datanode_2 | Length: 172 [application/octet-stream] kms_1 | Saving to: '/etc/security/keytabs/testuser2.keytab' kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 s3g_1 | Saving to: '/etc/security/keytabs/testuser2.keytab' scm_1 | Download s3g/scm@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab datanode_3 | --2019-10-07 08:08:25-- http://kdc:8081/keytab/d8b9c0e14ce8/s3g datanode_1 | recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. om_1 | Saving to: '/etc/security/keytabs/testuser2.keytab' datanode_2 | Saving to: '/etc/security/keytabs/testuser2.keytab' kms_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | scm_1 | --2019-10-07 08:08:25-- http://kdc:8081/keytab/scm/s3g datanode_3 | Resolving kdc (kdc)... 172.18.0.4 datanode_1 | 0K 100% 8.33M=0s recon_1 | HTTP request sent, awaiting response... 200 OK om_1 | kms_1 | 0K 100% 9.10M=0s datanode_2 | s3g_1 | 0K 100% 15.3M=0s kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_3 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_1 | recon_1 | Length: 158 [application/octet-stream] om_1 | 0K 100% 11.4M=0s kms_1 | datanode_2 | 0K 100% 10.9M=0s s3g_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_3 | HTTP request sent, awaiting response... 200 OK datanode_1 | 2019-10-07 08:08:25 (8.33 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [172/172] recon_1 | Saving to: '/etc/security/keytabs/testuser2.keytab' om_1 | kms_1 | 2019-10-07 08:08:25 (9.10 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [172/172] datanode_2 | s3g_1 | 2019-10-07 08:08:25 (15.3 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [154/154] kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 scm_1 | HTTP request sent, awaiting response... 200 OK datanode_3 | Length: 160 [application/octet-stream] datanode_1 | om_1 | 2019-10-07 08:08:24 (11.4 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [152/152] recon_1 | kms_1 | datanode_2 | 2019-10-07 08:08:24 (10.9 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [172/172] s3g_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | Length: 142 [application/octet-stream] datanode_3 | Saving to: '/etc/security/keytabs/s3g.keytab' om_1 | recon_1 | 0K 100% 11.6M=0s kms_1 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab datanode_2 | datanode_1 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab s3g_1 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | Saving to: '/etc/security/keytabs/s3g.keytab' datanode_3 | om_1 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab kms_1 | KVNO Timestamp Principal datanode_2 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab datanode_1 | KVNO Timestamp Principal recon_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | om_1 | KVNO Timestamp Principal datanode_3 | 0K 100% 12.9M=0s s3g_1 | KVNO Timestamp Principal kms_1 | ---- ----------------- -------------------------------------------------------- datanode_2 | KVNO Timestamp Principal datanode_1 | ---- ----------------- -------------------------------------------------------- recon_1 | 2019-10-07 08:08:43 (11.6 MB/s) - '/etc/security/keytabs/testuser2.keytab' saved [158/158] kdc_1 | Oct 07 08:08:20 kdc kadmind[15](info): closing down fd 18 scm_1 | 0K 100% 13.3M=0s om_1 | ---- ----------------- -------------------------------------------------------- datanode_3 | s3g_1 | ---- ----------------- -------------------------------------------------------- kms_1 | 2 10/07/19 08:08:25 testuser2/c8a6426e556a@EXAMPLE.COM datanode_2 | ---- ----------------- -------------------------------------------------------- datanode_1 | 2 10/07/19 08:08:25 testuser2/ad18a6fa7980@EXAMPLE.COM recon_1 | kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | om_1 | 2 10/07/19 08:08:24 testuser2/om@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:25 (12.9 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [160/160] s3g_1 | 2 10/07/19 08:08:25 testuser2/s3g@EXAMPLE.COM kms_1 | 2 10/07/19 08:08:25 testuser2/c8a6426e556a@EXAMPLE.COM datanode_2 | 2 10/07/19 08:08:24 testuser2/311de0fd64d4@EXAMPLE.COM datanode_1 | 2 10/07/19 08:08:25 testuser2/ad18a6fa7980@EXAMPLE.COM recon_1 | Keytab name: FILE:/etc/security/keytabs/testuser2.keytab kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:08:25 (13.3 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [142/142] datanode_3 | om_1 | 2 10/07/19 08:08:24 testuser2/om@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:25 testuser2/s3g@EXAMPLE.COM kms_1 | Download s3g/c8a6426e556a@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab datanode_2 | 2 10/07/19 08:08:24 testuser2/311de0fd64d4@EXAMPLE.COM datanode_1 | Download s3g/ad18a6fa7980@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab recon_1 | KVNO Timestamp Principal kdc_1 | Oct 07 08:08:20 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_3 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab om_1 | Download s3g/om@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab scm_1 | s3g_1 | Download s3g/s3g@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab kms_1 | --2019-10-07 08:08:25-- http://kdc:8081/keytab/c8a6426e556a/s3g datanode_2 | Download s3g/311de0fd64d4@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab datanode_1 | --2019-10-07 08:08:25-- http://kdc:8081/keytab/ad18a6fa7980/s3g recon_1 | ---- ----------------- -------------------------------------------------------- kdc_1 | Entry for principal dn/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.scm.keytab. datanode_3 | KVNO Timestamp Principal om_1 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/om/s3g scm_1 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab s3g_1 | --2019-10-07 08:08:25-- http://kdc:8081/keytab/s3g/s3g kms_1 | Resolving kdc (kdc)... 172.18.0.4 datanode_2 | --2019-10-07 08:08:24-- http://kdc:8081/keytab/311de0fd64d4/s3g datanode_1 | Resolving kdc (kdc)... 172.18.0.4 recon_1 | 2 10/07/19 08:08:43 testuser2/recon@EXAMPLE.COM kdc_1 | Entry for principal dn/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.scm.keytab. datanode_3 | ---- ----------------- -------------------------------------------------------- om_1 | Resolving kdc (kdc)... 172.18.0.4 s3g_1 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | KVNO Timestamp Principal kms_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | Resolving kdc (kdc)... 172.18.0.4 datanode_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. recon_1 | 2 10/07/19 08:08:43 testuser2/recon@EXAMPLE.COM kdc_1 | Generiting keytab datanode_3 | 2 10/07/19 08:08:25 s3g/d8b9c0e14ce8@EXAMPLE.COM om_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. s3g_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. scm_1 | ---- ----------------- -------------------------------------------------------- kms_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_1 | HTTP request sent, awaiting response... 200 OK recon_1 | Download s3g/recon@EXAMPLE.COM keytab file to /etc/security/keytabs/s3g.keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_3 | 2 10/07/19 08:08:25 s3g/d8b9c0e14ce8@EXAMPLE.COM om_1 | HTTP request sent, awaiting response... 200 OK s3g_1 | HTTP request sent, awaiting response... 200 OK scm_1 | 2 10/07/19 08:08:25 s3g/scm@EXAMPLE.COM kms_1 | Length: 160 [application/octet-stream] datanode_2 | HTTP request sent, awaiting response... 200 OK datanode_1 | Length: 160 [application/octet-stream] kdc_1 | WARNING: no policy specified for om/om@EXAMPLE.COM; defaulting to no policy datanode_3 | 2019-10-07 08:08:26 INFO HddsDatanodeService:51 - STARTUP_MSG: s3g_1 | Length: 142 [application/octet-stream] om_1 | Length: 140 [application/octet-stream] recon_1 | --2019-10-07 08:08:43-- http://kdc:8081/keytab/recon/s3g scm_1 | 2 10/07/19 08:08:25 s3g/scm@EXAMPLE.COM kms_1 | Saving to: '/etc/security/keytabs/s3g.keytab' datanode_2 | Length: 160 [application/octet-stream] datanode_1 | Saving to: '/etc/security/keytabs/s3g.keytab' kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_3 | /************************************************************ s3g_1 | Saving to: '/etc/security/keytabs/s3g.keytab' om_1 | Saving to: '/etc/security/keytabs/s3g.keytab' recon_1 | Resolving kdc (kdc)... 172.18.0.4 scm_1 | 2019-10-07 08:08:26 INFO StorageContainerManagerStarter:51 - STARTUP_MSG: datanode_2 | Saving to: '/etc/security/keytabs/s3g.keytab' datanode_1 | kms_1 | kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_3 | STARTUP_MSG: Starting HddsDatanodeService s3g_1 | om_1 | recon_1 | Connecting to kdc (kdc)|172.18.0.4|:8081... connected. datanode_2 | datanode_1 | 0K 100% 22.2M=0s kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database kms_1 | 0K 100% 21.4M=0s scm_1 | /************************************************************ datanode_3 | STARTUP_MSG: host = d8b9c0e14ce8/172.18.0.10 s3g_1 | 0K 100% 19.0M=0s om_1 | 0K 100% 13.0M=0s recon_1 | HTTP request sent, awaiting response... 200 OK datanode_2 | 0K 100% 12.6M=0s datanode_1 | kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM kms_1 | scm_1 | STARTUP_MSG: Starting StorageContainerManager s3g_1 | datanode_3 | STARTUP_MSG: args = [] om_1 | recon_1 | Length: 146 [application/octet-stream] datanode_2 | datanode_1 | 2019-10-07 08:08:25 (22.2 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [160/160] kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database kms_1 | 2019-10-07 08:08:25 (21.4 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [160/160] scm_1 | STARTUP_MSG: host = scm/172.18.0.7 s3g_1 | 2019-10-07 08:08:26 (19.0 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [142/142] datanode_3 | STARTUP_MSG: version = 3.2.0 om_1 | 2019-10-07 08:08:25 (13.0 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [140/140] recon_1 | Saving to: '/etc/security/keytabs/s3g.keytab' datanode_2 | 2019-10-07 08:08:25 (12.6 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [160/160] datanode_1 | kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM kms_1 | scm_1 | STARTUP_MSG: args = [--init] s3g_1 | datanode_3 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar om_1 | recon_1 | datanode_2 | datanode_1 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database kms_1 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab scm_1 | STARTUP_MSG: version = 3.2.0 s3g_1 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab om_1 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab datanode_3 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z recon_1 | 0K 100% 13.0M=0s datanode_2 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab datanode_1 | KVNO Timestamp Principal kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM kms_1 | KVNO Timestamp Principal s3g_1 | KVNO Timestamp Principal scm_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hamcrest-all-1.3.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar datanode_3 | STARTUP_MSG: java = 11.0.3 recon_1 | om_1 | KVNO Timestamp Principal datanode_2 | KVNO Timestamp Principal datanode_1 | ---- ----------------- -------------------------------------------------------- kms_1 | ---- ----------------- -------------------------------------------------------- kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | ---- ----------------- -------------------------------------------------------- scm_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_3 | ************************************************************/ recon_1 | 2019-10-07 08:08:43 (13.0 MB/s) - '/etc/security/keytabs/s3g.keytab' saved [146/146] om_1 | ---- ----------------- -------------------------------------------------------- datanode_2 | ---- ----------------- -------------------------------------------------------- datanode_1 | 2 10/07/19 08:08:25 s3g/ad18a6fa7980@EXAMPLE.COM kms_1 | 2 10/07/19 08:08:25 s3g/c8a6426e556a@EXAMPLE.COM kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | 2 10/07/19 08:08:26 s3g/s3g@EXAMPLE.COM scm_1 | STARTUP_MSG: java = 11.0.3 datanode_3 | 2019-10-07 08:08:26 INFO HddsDatanodeService:51 - registered UNIX signal handlers for [TERM, HUP, INT] om_1 | 2 10/07/19 08:08:25 s3g/om@EXAMPLE.COM recon_1 | datanode_2 | 2 10/07/19 08:08:25 s3g/311de0fd64d4@EXAMPLE.COM datanode_1 | 2 10/07/19 08:08:25 s3g/ad18a6fa7980@EXAMPLE.COM kms_1 | 2 10/07/19 08:08:25 s3g/c8a6426e556a@EXAMPLE.COM kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | 2 10/07/19 08:08:26 s3g/s3g@EXAMPLE.COM scm_1 | ************************************************************/ datanode_3 | 2019-10-07 08:08:27 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties om_1 | 2 10/07/19 08:08:25 s3g/om@EXAMPLE.COM recon_1 | Keytab name: FILE:/etc/security/keytabs/s3g.keytab datanode_2 | 2 10/07/19 08:08:25 s3g/311de0fd64d4@EXAMPLE.COM kms_1 | # Licensed to the Apache Software Foundation (ASF) under one or more datanode_1 | 2019-10-07 08:08:26 INFO HddsDatanodeService:51 - STARTUP_MSG: kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | WARNING: An illegal reflective access operation has occurred scm_1 | 2019-10-07 08:08:26 INFO StorageContainerManagerStarter:51 - registered UNIX signal handlers for [TERM, HUP, INT] datanode_3 | 2019-10-07 08:08:27 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). om_1 | 2019-10-07 08:08:26 INFO OzoneManagerStarter:51 - STARTUP_MSG: recon_1 | KVNO Timestamp Principal datanode_2 | 2019-10-07 08:08:26 INFO HddsDatanodeService:51 - STARTUP_MSG: kms_1 | # contributor license agreements. See the NOTICE file distributed with datanode_1 | /************************************************************ kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() scm_1 | 2019-10-07 08:08:27 WARN ServerUtils:145 - ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_3 | 2019-10-07 08:08:27 INFO MetricsSystemImpl:191 - HddsDatanode metrics system started om_1 | /************************************************************ recon_1 | ---- ----------------- -------------------------------------------------------- datanode_2 | /************************************************************ kms_1 | # this work for additional information regarding copyright ownership. datanode_1 | STARTUP_MSG: Starting HddsDatanodeService kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil scm_1 | SCM initialization succeeded.Current cluster id for sd=/data/metadata/scm;cid=CID-ebaad74e-8246-48b0-9671-02b0ae8169f9 om_1 | STARTUP_MSG: Starting OzoneManager datanode_3 | 2019-10-07 08:08:27 INFO HddsDatanodeService:184 - HddsDatanodeService host:d8b9c0e14ce8 ip:172.18.0.10 datanode_2 | STARTUP_MSG: Starting HddsDatanodeService kms_1 | # The ASF licenses this file to You under the Apache License, Version 2.0 datanode_1 | STARTUP_MSG: host = ad18a6fa7980/172.18.0.8 kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations recon_1 | 2 10/07/19 08:08:43 s3g/recon@EXAMPLE.COM scm_1 | 2019-10-07 08:08:27 INFO StorageContainerManagerStarter:51 - SHUTDOWN_MSG: om_1 | STARTUP_MSG: host = om/172.18.0.2 datanode_3 | 2019-10-07 08:08:28 INFO HddsDatanodeService:194 - Ozone security is enabled. Attempting login for Hdds Datanode user. Principal: dn/_HOST@EXAMPLE.COM,keytab: /etc/security/keytabs/dn.keytab datanode_2 | STARTUP_MSG: host = 311de0fd64d4/172.18.0.5 kms_1 | # (the "License"); you may not use this file except in compliance with datanode_1 | STARTUP_MSG: args = [] kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM recon_1 | 2 10/07/19 08:08:43 s3g/recon@EXAMPLE.COM om_1 | STARTUP_MSG: args = [--init] s3g_1 | WARNING: All illegal access operations will be denied in a future release datanode_3 | WARNING: An illegal reflective access operation has occurred datanode_2 | STARTUP_MSG: args = [] scm_1 | /************************************************************ kms_1 | # the License. You may obtain a copy of the License at kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database recon_1 | WARNING: An illegal reflective access operation has occurred datanode_1 | STARTUP_MSG: version = 3.2.0 om_1 | STARTUP_MSG: version = 3.2.0 s3g_1 | 2019-10-07 08:08:27 INFO DFSUtil:1629 - Starting web server as: HTTP/s3g@EXAMPLE.COM datanode_3 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() scm_1 | SHUTDOWN_MSG: Shutting down StorageContainerManager at scm/172.18.0.7 kms_1 | # kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM recon_1 | WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/opt/hadoop/share/ozone/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) datanode_2 | STARTUP_MSG: version = 3.2.0 datanode_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar s3g_1 | 2019-10-07 08:08:27 INFO DFSUtil:1641 - Starting Web-server for s3gateway at: http://0.0.0.0:9878 datanode_3 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil om_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/sqlite-jdbc-3.25.2.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-tools-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar scm_1 | ************************************************************/ kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database recon_1 | WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2 datanode_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_2 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar datanode_3 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations recon_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z scm_1 | 2019-10-07 08:08:28 INFO StorageContainerManagerStarter:51 - STARTUP_MSG: datanode_1 | STARTUP_MSG: java = 11.0.3 kdc_1 | Oct 07 08:08:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435700, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM kms_1 | # http://www.apache.org/licenses/LICENSE-2.0 datanode_2 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_3 | WARNING: All illegal access operations will be denied in a future release recon_1 | WARNING: All illegal access operations will be denied in a future release s3g_1 | 2019-10-07 08:08:27 INFO log:192 - Logging initialized @1319ms scm_1 | /************************************************************ kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_1 | ************************************************************/ om_1 | STARTUP_MSG: java = 11.0.3 kms_1 | # datanode_2 | STARTUP_MSG: java = 11.0.3 datanode_3 | 2019-10-07 08:08:28 INFO UserGroupInformation:1009 - Login successful for user dn/d8b9c0e14ce8@EXAMPLE.COM using keytab file /etc/security/keytabs/dn.keytab recon_1 | 2019-10-07 08:08:44 INFO ReconRestServletModule:75 - rest([/api/*]).packages(org.apache.hadoop.ozone.recon.api) s3g_1 | 2019-10-07 08:08:27 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. scm_1 | STARTUP_MSG: Starting StorageContainerManager kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_1 | 2019-10-07 08:08:26 INFO HddsDatanodeService:51 - registered UNIX signal handlers for [TERM, HUP, INT] om_1 | ************************************************************/ kms_1 | # Unless required by applicable law or agreed to in writing, software datanode_2 | ************************************************************/ datanode_3 | 2019-10-07 08:08:28 INFO HddsDatanodeService:208 - Hdds Datanode login successful. s3g_1 | 2019-10-07 08:08:27 INFO HttpRequestLog:81 - Http request log for http.requests.s3gateway is not defined recon_1 | 2019-10-07 08:08:45 INFO ReconServer:74 - Initializing Recon server... scm_1 | STARTUP_MSG: host = scm/172.18.0.7 kdc_1 | Principal "om/om@EXAMPLE.COM" created. om_1 | 2019-10-07 08:08:26 INFO OzoneManagerStarter:51 - registered UNIX signal handlers for [TERM, HUP, INT] kms_1 | # distributed under the License is distributed on an "AS IS" BASIS, datanode_1 | 2019-10-07 08:08:27 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties datanode_2 | 2019-10-07 08:08:26 INFO HddsDatanodeService:51 - registered UNIX signal handlers for [TERM, HUP, INT] datanode_3 | 2019-10-07 08:08:28 INFO HddsDatanodeService:239 - Initializing secure Datanode. s3g_1 | 2019-10-07 08:08:27 INFO HttpServer2:975 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) recon_1 | 2019-10-07 08:08:45 ERROR ReconServer:99 - Error during initializing Recon server. scm_1 | STARTUP_MSG: args = [] kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:08:27 INFO OMHANodeDetails:190 - Configuration either no ozone.om.address set. Falling back to the default OM address om/172.18.0.2:9862 kms_1 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. datanode_1 | 2019-10-07 08:08:27 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). datanode_2 | 2019-10-07 08:08:26 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties datanode_3 | 2019-10-07 08:08:28 ERROR DNCertificateClient:228 - Default certificate serial id is not set. Can't locate the default certificate for this client. s3g_1 | 2019-10-07 08:08:27 INFO HttpServer2:948 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context s3gateway recon_1 | java.sql.SQLException: path to '//data/metadata/recon/ozone_recon_sqlite.db': '/data/metadata' does not exist scm_1 | STARTUP_MSG: version = 3.2.0 kdc_1 | Entry for principal om/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.om.keytab. om_1 | 2019-10-07 08:08:27 INFO OMHANodeDetails:220 - OM Service ID is not set. Setting it to the default ID: omServiceIdDefault kms_1 | # See the License for the specific language governing permissions and datanode_1 | 2019-10-07 08:08:27 INFO MetricsSystemImpl:191 - HddsDatanode metrics system started datanode_2 | 2019-10-07 08:08:26 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). datanode_3 | 2019-10-07 08:08:28 INFO DNCertificateClient:641 - Certificate client init case: 0 recon_1 | at org.sqlite.SQLiteConnection.open(SQLiteConnection.java:215) kdc_1 | Entry for principal om/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.om.keytab. scm_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hamcrest-all-1.3.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar om_1 | WARNING: An illegal reflective access operation has occurred kms_1 | # limitations under the License. datanode_1 | 2019-10-07 08:08:27 INFO HddsDatanodeService:184 - HddsDatanodeService host:ad18a6fa7980 ip:172.18.0.8 datanode_2 | 2019-10-07 08:08:26 INFO MetricsSystemImpl:191 - HddsDatanode metrics system started datanode_3 | 2019-10-07 08:08:28 INFO DNCertificateClient:655 - Creating keypair for client as keypair and certificate not found. s3g_1 | 2019-10-07 08:08:27 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs recon_1 | at org.sqlite.SQLiteConnection.(SQLiteConnection.java:61) kdc_1 | Generiting keytab om_1 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() kms_1 | scm_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_2 | 2019-10-07 08:08:26 INFO HddsDatanodeService:184 - HddsDatanodeService host:311de0fd64d4 ip:172.18.0.5 datanode_3 | 2019-10-07 08:08:28 INFO HddsDatanodeService:242 - Init response: GETCERT s3g_1 | 2019-10-07 08:08:27 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static recon_1 | at org.sqlite.jdbc3.JDBC3Connection.(JDBC3Connection.java:28) datanode_1 | 2019-10-07 08:08:27 INFO HddsDatanodeService:194 - Ozone security is enabled. Attempting login for Hdds Datanode user. Principal: dn/_HOST@EXAMPLE.COM,keytab: /etc/security/keytabs/dn.keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil kms_1 | [logging] scm_1 | STARTUP_MSG: java = 11.0.3 datanode_3 | 2019-10-07 08:08:28 INFO OzoneSecurityUtil:103 - Adding ip:172.18.0.10,host:d8b9c0e14ce8 s3g_1 | 2019-10-07 08:08:27 DEBUG HttpServer2:857 - Found existing conf servlet at path /conf; will replace mapping with conf servlet datanode_1 | WARNING: An illegal reflective access operation has occurred datanode_2 | 2019-10-07 08:08:27 INFO HddsDatanodeService:194 - Ozone security is enabled. Attempting login for Hdds Datanode user. Principal: dn/_HOST@EXAMPLE.COM,keytab: /etc/security/keytabs/dn.keytab kdc_1 | WARNING: no policy specified for om/311de0fd64d4@EXAMPLE.COM; defaulting to no policy om_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations kms_1 | default = FILE:/var/log/krb5libs.log recon_1 | at org.sqlite.jdbc4.JDBC4Connection.(JDBC4Connection.java:21) scm_1 | ************************************************************/ datanode_3 | 2019-10-07 08:08:28 INFO OzoneSecurityUtil:107 - ip:127.0.0.1,host:localhost not returned. s3g_1 | 2019-10-07 08:08:27 INFO Gateway:57 - Starting Ozone S3 gateway datanode_1 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() kdc_1 | Principal "om/311de0fd64d4@EXAMPLE.COM" created. om_1 | WARNING: All illegal access operations will be denied in a future release kms_1 | kdc = FILE:/var/log/krb5kdc.log datanode_2 | WARNING: An illegal reflective access operation has occurred recon_1 | at org.sqlite.JDBC.createConnection(JDBC.java:116) scm_1 | 2019-10-07 08:08:28 INFO StorageContainerManagerStarter:51 - registered UNIX signal handlers for [TERM, HUP, INT] datanode_3 | 2019-10-07 08:08:28 INFO HddsDatanodeService:318 - Creating csr for DN-> subject:root@d8b9c0e14ce8 s3g_1 | 2019-10-07 08:08:27 DEBUG HttpServer2:1273 - opening listeners: [ServerConnector@dc9876b{HTTP/1.1,[http/1.1]}{0.0.0.0:9878}] datanode_1 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:08:27 INFO UserGroupInformation:1009 - Login successful for user om/om@EXAMPLE.COM using keytab file /etc/security/keytabs/om.keytab kms_1 | admin_server = FILE:/var/log/kadmind.log datanode_2 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() recon_1 | at org.sqlite.SQLiteDataSource.getConnection(SQLiteDataSource.java:410) scm_1 | 2019-10-07 08:08:28 WARN ServerUtils:145 - ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. s3g_1 | 2019-10-07 08:08:27 INFO HttpServer2:1191 - Jetty bound to port 9878 datanode_3 | 2019-10-07 08:08:30 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9961. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=2147483647, sleepTime=1000 MILLISECONDS) datanode_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations kdc_1 | Entry for principal om/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.311de0fd64d4.keytab. om_1 | 2019-10-07 08:08:27 INFO OzoneManager:752 - Ozone Manager login successful. kms_1 | datanode_2 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil recon_1 | at org.sqlite.SQLiteDataSource.getConnection(SQLiteDataSource.java:398) s3g_1 | 2019-10-07 08:08:27 INFO Server:351 - jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_3 | 2019-10-07 08:08:32 INFO HddsDatanodeService:249 - Successfully stored SCM signed certificate, case:GETCERT. kdc_1 | Entry for principal om/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.311de0fd64d4.keytab. datanode_1 | WARNING: All illegal access operations will be denied in a future release scm_1 | WARNING: An illegal reflective access operation has occurred om_1 | 2019-10-07 08:08:27 WARN ServerUtils:222 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. kms_1 | [libdefaults] datanode_2 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations recon_1 | at org.hadoop.ozone.recon.schema.StatsSchemaDefinition.initializeSchema(StatsSchemaDefinition.java:44) s3g_1 | 2019-10-07 08:08:27 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_3 | 2019-10-07 08:08:32 INFO HddsVolume:176 - Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 5454080942080 kdc_1 | Generiting keytab datanode_1 | 2019-10-07 08:08:28 INFO UserGroupInformation:1009 - Login successful for user dn/ad18a6fa7980@EXAMPLE.COM using keytab file /etc/security/keytabs/dn.keytab scm_1 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() om_1 | 2019-10-07 08:08:28 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9863. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) kms_1 | dns_canonicalize_hostname = false datanode_2 | WARNING: All illegal access operations will be denied in a future release recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:78) s3g_1 | 2019-10-07 08:08:27 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/s3g@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:32 INFO VolumeSet:170 - Added Volume : /data/hdds/hdds to VolumeSet kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_1 | 2019-10-07 08:08:28 INFO HddsDatanodeService:208 - Hdds Datanode login successful. scm_1 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil om_1 | 2019-10-07 08:08:29 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9863. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) kms_1 | dns_lookup_realm = false datanode_2 | 2019-10-07 08:08:27 INFO UserGroupInformation:1009 - Login successful for user dn/311de0fd64d4@EXAMPLE.COM using keytab file /etc/security/keytabs/dn.keytab recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:41) s3g_1 | 2019-10-07 08:08:27 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@5c86dbc5{/logs,file:///var/log/hadoop/,AVAILABLE} datanode_3 | 2019-10-07 08:08:32 INFO ThrottledAsyncChecker:139 - Scheduling a check for org.apache.hadoop.ozone.container.common.volume.HddsVolume@537c8c7e datanode_1 | 2019-10-07 08:08:28 INFO HddsDatanodeService:239 - Initializing secure Datanode. kdc_1 | WARNING: no policy specified for dn/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy scm_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations om_1 | 2019-10-07 08:08:30 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9863. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) kms_1 | ticket_lifetime = 24h datanode_2 | 2019-10-07 08:08:27 INFO HddsDatanodeService:208 - Hdds Datanode login successful. recon_1 | at picocli.CommandLine.execute(CommandLine.java:1173) s3g_1 | 2019-10-07 08:08:27 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@769a1df5{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-ozone-s3gateway-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} datanode_3 | 2019-10-07 08:08:32 INFO HddsVolumeChecker:202 - Scheduled health check for volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@537c8c7e datanode_1 | 2019-10-07 08:08:28 ERROR DNCertificateClient:228 - Default certificate serial id is not set. Can't locate the default certificate for this client. kdc_1 | Principal "dn/ad18a6fa7980@EXAMPLE.COM" created. scm_1 | WARNING: All illegal access operations will be denied in a future release om_1 | 2019-10-07 08:08:31 INFO OzoneManager:934 - Initializing secure OzoneManager. kms_1 | renew_lifetime = 7d datanode_2 | 2019-10-07 08:08:27 INFO HddsDatanodeService:239 - Initializing secure Datanode. recon_1 | at picocli.CommandLine.access$800(CommandLine.java:141) s3g_1 | 2019-10-07 08:08:28,728 | INFO | servletWeldServlet | WELD-ENV-001007: Initialize Weld using ServletContextListener | datanode_3 | 2019-10-07 08:08:32 WARN HddsServerUtil:354 - Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs datanode_1 | 2019-10-07 08:08:28 INFO DNCertificateClient:641 - Certificate client init case: 0 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:08:28 INFO UserGroupInformation:1009 - Login successful for user scm/scm@EXAMPLE.COM using keytab file /etc/security/keytabs/scm.keytab om_1 | 2019-10-07 08:08:32 ERROR OMCertificateClient:228 - Default certificate serial id is not set. Can't locate the default certificate for this client. kms_1 | forwardable = true datanode_2 | 2019-10-07 08:08:27 ERROR DNCertificateClient:228 - Default certificate serial id is not set. Can't locate the default certificate for this client. recon_1 | at picocli.CommandLine$RunLast.handle(CommandLine.java:1367) s3g_1 | 2019-10-07 08:08:28,750 | INFO | Version | WELD-000900: 2.4.7 (Final) | datanode_3 | 2019-10-07 08:08:32 INFO RaftServerProxy:43 - raft.rpc.type = GRPC (default) datanode_1 | 2019-10-07 08:08:28 INFO DNCertificateClient:655 - Creating keypair for client as keypair and certificate not found. kdc_1 | Entry for principal dn/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.ad18a6fa7980.keytab. scm_1 | 2019-10-07 08:08:28 INFO StorageContainerManager:514 - SCM login successful. om_1 | 2019-10-07 08:08:32 INFO OMCertificateClient:641 - Certificate client init case: 0 kms_1 | rdns = false datanode_2 | 2019-10-07 08:08:27 INFO DNCertificateClient:641 - Certificate client init case: 0 s3g_1 | 2019-10-07 08:08:29,097 | INFO | Bootstrap | WELD-000101: Transactional services not available. Injection of @Inject UserTransaction not available. Transactional observers will be invoked synchronously. | datanode_3 | 2019-10-07 08:08:32 INFO GrpcConfigKeys$Server:43 - raft.grpc.server.port = 9858 (custom) recon_1 | at picocli.CommandLine$RunLast.handle(CommandLine.java:1335) datanode_1 | 2019-10-07 08:08:28 INFO HddsDatanodeService:242 - Init response: GETCERT kdc_1 | Entry for principal dn/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.ad18a6fa7980.keytab. scm_1 | 2019-10-07 08:08:28 WARN ServerUtils:145 - ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. om_1 | 2019-10-07 08:08:32 INFO OMCertificateClient:57 - Creating keypair for client as keypair and certificate not found. kms_1 | default_realm = EXAMPLE.COM datanode_2 | 2019-10-07 08:08:27 INFO DNCertificateClient:655 - Creating keypair for client as keypair and certificate not found. s3g_1 | 2019-10-07 08:08:29,272 | INFO | Event | WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds. | datanode_3 | 2019-10-07 08:08:32 INFO GrpcService:43 - raft.grpc.message.size.max = 33570816 (custom) recon_1 | at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) datanode_1 | 2019-10-07 08:08:28 INFO OzoneSecurityUtil:103 - Adding ip:172.18.0.8,host:ad18a6fa7980 kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:08:28 INFO log:192 - Logging initialized @1458ms om_1 | 2019-10-07 08:08:32 INFO OzoneManager:940 - Init response: GETCERT datanode_2 | 2019-10-07 08:08:28 INFO HddsDatanodeService:242 - Init response: GETCERT s3g_1 | 2019-10-07 08:08:29,498 | INFO | servletJetty | WELD-ENV-001201: Jetty 7.2+ detected, CDI injection will be available in Servlets and Filters. Injection into Listeners is not supported. | datanode_3 | 2019-10-07 08:08:32 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) recon_1 | at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:117 - using custom profile for table: deletedBlocks datanode_1 | 2019-10-07 08:08:28 INFO OzoneSecurityUtil:107 - ip:127.0.0.1,host:localhost not returned. kms_1 | om_1 | 2019-10-07 08:08:33 INFO OzoneSecurityUtil:103 - Adding ip:172.18.0.2,host:om datanode_2 | 2019-10-07 08:08:28 INFO OzoneSecurityUtil:103 - Adding ip:172.18.0.5,host:311de0fd64d4 s3g_1 | 2019-10-07 08:08:29 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/s3g@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:32 INFO GrpcService:43 - raft.grpc.flow.control.window = 1MB (=1048576) (default) recon_1 | at picocli.CommandLine.parseWithHandler(CommandLine.java:1465) kdc_1 | WARNING: no policy specified for dn/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:deletedBlocks datanode_1 | 2019-10-07 08:08:28 INFO HddsDatanodeService:318 - Creating csr for DN-> subject:root@ad18a6fa7980 kms_1 | [realms] om_1 | 2019-10-07 08:08:33 INFO OzoneSecurityUtil:107 - ip:127.0.0.1,host:localhost not returned. datanode_2 | 2019-10-07 08:08:28 INFO OzoneSecurityUtil:107 - ip:127.0.0.1,host:localhost not returned. s3g_1 | Oct 07, 2019 8:08:30 AM org.glassfish.jersey.internal.Errors logErrors datanode_3 | 2019-10-07 08:08:32 INFO RaftServerConfigKeys:43 - raft.server.rpc.request.timeout = 3000ms (default) recon_1 | at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65) kdc_1 | Principal "dn/d8b9c0e14ce8@EXAMPLE.COM" created. scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:117 - using custom profile for table: validCerts kms_1 | EXAMPLE.COM = { datanode_1 | 2019-10-07 08:08:29 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9961. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=2147483647, sleepTime=1000 MILLISECONDS) om_1 | 2019-10-07 08:08:33 INFO OzoneManager:1354 - Creating csr for OM->dns:om,ip:172.18.0.2,scmId:47fb8ae2-8429-4ba3-9c4b-9cb0716b9201,clusterId:CID-ebaad74e-8246-48b0-9671-02b0ae8169f9,subject:root@om datanode_2 | 2019-10-07 08:08:28 INFO HddsDatanodeService:318 - Creating csr for DN-> subject:root@311de0fd64d4 s3g_1 | WARNING: The following warnings have been detected: WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.get(java.lang.String,java.lang.String,java.lang.String,int,java.lang.String,java.io.InputStream) throws java.io.IOException,org.apache.hadoop.ozone.s3.exception.OS3Exception, should not consume any entity. datanode_3 | 2019-10-07 08:08:33 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. recon_1 | at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56) scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:validCerts kms_1 | kdc = kdc datanode_1 | 2019-10-07 08:08:30 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9961. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=2147483647, sleepTime=1000 MILLISECONDS) datanode_2 | 2019-10-07 08:08:29 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9961. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=2147483647, sleepTime=1000 MILLISECONDS) om_1 | 2019-10-07 08:08:33 INFO OzoneManager:1371 - OzoneManager ports added:[name: "RPC" s3g_1 | datanode_3 | 2019-10-07 08:08:33 INFO DFSUtil:1629 - Starting web server as: HTTP/d8b9c0e14ce8@EXAMPLE.COM kdc_1 | Entry for principal dn/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.d8b9c0e14ce8.keytab. recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:52) kms_1 | admin_server = kdc datanode_1 | 2019-10-07 08:08:32 INFO HddsDatanodeService:249 - Successfully stored SCM signed certificate, case:GETCERT. om_1 | value: 9862 datanode_2 | 2019-10-07 08:08:30 INFO Client:948 - Retrying connect to server: scm/172.18.0.7:9961. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=2147483647, sleepTime=1000 MILLISECONDS) scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:117 - using custom profile for table: revokedCerts s3g_1 | 2019-10-07 08:08:30 INFO ContextHandler:781 - Started o.e.j.w.WebAppContext@3830f918{/,file:///tmp/jetty-0.0.0.0-9878-s3gateway-_-any-4391445863104282534.dir/webapp/,AVAILABLE}{/s3gateway} datanode_3 | 2019-10-07 08:08:33 INFO DFSUtil:1641 - Starting Web-server for hddsDatanode at: http://0.0.0.0:9882 kdc_1 | Entry for principal dn/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.d8b9c0e14ce8.keytab. recon_1 | 2019-10-07 08:08:45 INFO ReconServer:114 - Stopping Recon server datanode_1 | 2019-10-07 08:08:32 INFO HddsVolume:176 - Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 5454080942080 kms_1 | } om_1 | ] datanode_2 | 2019-10-07 08:08:32 INFO HddsDatanodeService:249 - Successfully stored SCM signed certificate, case:GETCERT. scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:revokedCerts s3g_1 | 2019-10-07 08:08:30 INFO AbstractConnector:278 - Started ServerConnector@dc9876b{HTTP/1.1,[http/1.1]}{0.0.0.0:9878} kdc_1 | Generiting keytab datanode_3 | 2019-10-07 08:08:33 INFO log:192 - Logging initialized @7046ms recon_1 | java.lang.NullPointerException datanode_1 | 2019-10-07 08:08:32 INFO VolumeSet:170 - Added Volume : /data/hdds/hdds to VolumeSet kms_1 | om_1 | 2019-10-07 08:08:33 INFO OzoneManager:947 - Successfully stored SCM signed certificate. datanode_2 | 2019-10-07 08:08:32 INFO HddsVolume:176 - Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 5454080942080 scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:117 - using custom profile for table: default s3g_1 | 2019-10-07 08:08:30 INFO Server:419 - Started @4343ms kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_3 | 2019-10-07 08:08:33 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.stop(ReconServer.java:115) datanode_1 | 2019-10-07 08:08:32 INFO ThrottledAsyncChecker:139 - Scheduling a check for org.apache.hadoop.ozone.container.common.volume.HddsVolume@6601cc93 kms_1 | [domain_realm] om_1 | OM initialization succeeded.Current cluster id for sd=/data/metadata/om;cid=CID-ebaad74e-8246-48b0-9671-02b0ae8169f9 datanode_2 | 2019-10-07 08:08:32 INFO VolumeSet:170 - Added Volume : /data/hdds/hdds to VolumeSet scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:174 - Using default column profile:DBProfile.DISK for Table:default kdc_1 | WARNING: no policy specified for dn/c8a6426e556a@EXAMPLE.COM; defaulting to no policy s3g_1 | 2019-10-07 08:08:30 INFO BaseHttpServer:215 - HTTP server of S3GATEWAY is listening at http://0.0.0.0:9878 datanode_3 | 2019-10-07 08:08:33 INFO HttpRequestLog:81 - Http request log for http.requests.hddsDatanode is not defined recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:100) datanode_1 | 2019-10-07 08:08:32 INFO HddsVolumeChecker:202 - Scheduled health check for volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@6601cc93 kms_1 | .example.com = EXAMPLE.COM om_1 | 2019-10-07 08:08:33 INFO OzoneManagerStarter:51 - SHUTDOWN_MSG: datanode_2 | 2019-10-07 08:08:32 INFO ThrottledAsyncChecker:139 - Scheduling a check for org.apache.hadoop.ozone.container.common.volume.HddsVolume@6601cc93 scm_1 | 2019-10-07 08:08:29 INFO DBStoreBuilder:205 - Using default options. DBProfile.DISK kdc_1 | Principal "dn/c8a6426e556a@EXAMPLE.COM" created. s3g_1 | 2019-10-07 08:15:43 INFO BucketEndpoint:211 - Location is /bucket-test123 datanode_3 | 2019-10-07 08:08:33 INFO HttpServer2:975 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:41) datanode_1 | 2019-10-07 08:08:32 WARN HddsServerUtil:354 - Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs kms_1 | WARNING: /opt/hadoop/temp does not exist. Creating. om_1 | /************************************************************ datanode_2 | 2019-10-07 08:08:32 INFO HddsVolumeChecker:202 - Scheduled health check for volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@6601cc93 scm_1 | 2019-10-07 08:08:30 INFO CallQueueManager:84 - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 200, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 2019-10-07 08:15:46 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_3 | 2019-10-07 08:08:33 INFO HttpServer2:948 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context hddsDatanode recon_1 | at picocli.CommandLine.execute(CommandLine.java:1173) datanode_1 | 2019-10-07 08:08:32 INFO RaftServerProxy:43 - raft.rpc.type = GRPC (default) om_1 | SHUTDOWN_MSG: Shutting down OzoneManager at om/172.18.0.2 kms_1 | WARNING: /opt/hadoop/logs does not exist. Creating. datanode_2 | 2019-10-07 08:08:32 WARN HddsServerUtil:354 - Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs scm_1 | 2019-10-07 08:08:30 INFO Server:1074 - Starting Socket Reader #1 for port 9961 kdc_1 | Entry for principal dn/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.c8a6426e556a.keytab. s3g_1 | 20191007T081546Z datanode_3 | 2019-10-07 08:08:33 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs recon_1 | at picocli.CommandLine.access$800(CommandLine.java:141) datanode_1 | 2019-10-07 08:08:32 INFO GrpcConfigKeys$Server:43 - raft.grpc.server.port = 9858 (custom) om_1 | ************************************************************/ kms_1 | Oct 07, 2019 8:08:28 AM com.sun.jersey.api.core.PackagesResourceConfig init scm_1 | 2019-10-07 08:08:30 INFO NodeSchemaLoader:125 - Loading file from java.lang.CompoundEnumeration@4d4960c8 kdc_1 | Entry for principal dn/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.c8a6426e556a.keytab. s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:33 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static recon_1 | at picocli.CommandLine$RunLast.handle(CommandLine.java:1367) datanode_1 | 2019-10-07 08:08:32 INFO GrpcService:43 - raft.grpc.message.size.max = 33570816 (custom) om_1 | 2019-10-07 08:08:34 INFO OzoneManagerStarter:51 - STARTUP_MSG: kms_1 | INFO: Scanning for root resource and provider classes in the packages: scm_1 | 2019-10-07 08:08:30 INFO NodeSchemaLoader:171 - Loading network topology layer schema file datanode_2 | 2019-10-07 08:08:32 INFO RaftServerProxy:43 - raft.rpc.type = GRPC (default) kdc_1 | Generiting keytab s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_3 | 2019-10-07 08:08:33 DEBUG HttpServer2:857 - Found existing conf servlet at path /conf; will replace mapping with conf servlet recon_1 | at picocli.CommandLine$RunLast.handle(CommandLine.java:1335) datanode_1 | 2019-10-07 08:08:32 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) om_1 | /************************************************************ kms_1 | org.apache.hadoop.crypto.key.kms.server scm_1 | 2019-10-07 08:08:30 INFO SCMNodeManager:116 - Entering startup safe mode. datanode_2 | 2019-10-07 08:08:32 INFO GrpcConfigKeys$Server:43 - raft.grpc.server.port = 9858 (custom) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 2019-10-07 08:15:46 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_3 | 2019-10-07 08:08:33 DEBUG HttpServer2:1273 - opening listeners: [ServerConnector@285005b8{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}] recon_1 | at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243) datanode_1 | 2019-10-07 08:08:32 INFO GrpcService:43 - raft.grpc.flow.control.window = 1MB (=1048576) (default) om_1 | STARTUP_MSG: Starting OzoneManager kms_1 | Oct 07, 2019 8:08:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses scm_1 | 2019-10-07 08:08:30 INFO ContainerPlacementPolicyFactory:57 - Create container placement policy of type org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom datanode_2 | 2019-10-07 08:08:32 INFO GrpcService:43 - raft.grpc.message.size.max = 33570816 (custom) kdc_1 | WARNING: no policy specified for dn/s3g@EXAMPLE.COM; defaulting to no policy s3g_1 | 20191007T081546Z datanode_3 | 2019-10-07 08:08:33 INFO HttpServer2:1191 - Jetty bound to port 9882 recon_1 | at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526) datanode_1 | 2019-10-07 08:08:32 INFO RaftServerConfigKeys:43 - raft.server.rpc.request.timeout = 3000ms (default) om_1 | STARTUP_MSG: host = om/172.18.0.2 kms_1 | INFO: Root resource classes found: scm_1 | 2019-10-07 08:08:30 WARN ServerUtils:145 - ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_2 | 2019-10-07 08:08:32 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) kdc_1 | Principal "dn/s3g@EXAMPLE.COM" created. s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:33 INFO Server:351 - jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 recon_1 | at picocli.CommandLine.parseWithHandler(CommandLine.java:1465) datanode_1 | 2019-10-07 08:08:33 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) om_1 | STARTUP_MSG: args = [] kms_1 | class org.apache.hadoop.crypto.key.kms.server.KMS scm_1 | 2019-10-07 08:08:30 INFO SCMPipelineManager:132 - No pipeline exists in current db datanode_2 | 2019-10-07 08:08:32 INFO GrpcService:43 - raft.grpc.flow.control.window = 1MB (=1048576) (default) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_3 | 2019-10-07 08:08:33 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. recon_1 | at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:65) datanode_1 | 2019-10-07 08:08:33 INFO DFSUtil:1629 - Starting web server as: HTTP/ad18a6fa7980@EXAMPLE.COM om_1 | STARTUP_MSG: version = 3.2.0 kms_1 | Oct 07, 2019 8:08:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses datanode_2 | 2019-10-07 08:08:32 INFO RaftServerConfigKeys:43 - raft.server.rpc.request.timeout = 3000ms (default) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 s3g_1 | 2019-10-07 08:15:46 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_3 | 2019-10-07 08:08:33 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/d8b9c0e14ce8@EXAMPLE.COM recon_1 | at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:56) datanode_1 | 2019-10-07 08:08:33 INFO DFSUtil:1641 - Starting Web-server for hddsDatanode at: http://0.0.0.0:9882 scm_1 | 2019-10-07 08:08:30 WARN ServerUtils:145 - ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. kms_1 | INFO: Provider classes found: datanode_2 | 2019-10-07 08:08:33 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) om_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/sqlite-jdbc-3.25.2.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-tools-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-201fc85-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar s3g_1 | 20191007T081546Z datanode_3 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@3f9b7fe1{/logs,file:///var/log/hadoop/,AVAILABLE} recon_1 | at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:52) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_1 | 2019-10-07 08:08:33 INFO log:192 - Logging initialized @7125ms scm_1 | 2019-10-07 08:08:30 WARN EventQueue:183 - No event handler registered for event TypedEvent{payloadType=SafeModeStatus, name='SafeModeStatus'} kms_1 | class org.apache.hadoop.crypto.key.kms.server.KMSJSONWriter datanode_2 | 2019-10-07 08:08:33 INFO DFSUtil:1629 - Starting web server as: HTTP/311de0fd64d4@EXAMPLE.COM om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@7d42404e{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:33 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. scm_1 | 2019-10-07 08:08:30 INFO CallQueueManager:84 - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. kms_1 | class org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider datanode_2 | 2019-10-07 08:08:33 INFO DFSUtil:1641 - Starting Web-server for hddsDatanode at: http://0.0.0.0:9882 om_1 | STARTUP_MSG: java = 11.0.3 s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover immediately. datanode_3 | 2019-10-07 08:08:33 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/d8b9c0e14ce8@EXAMPLE.COM kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:33 INFO HttpRequestLog:81 - Http request log for http.requests.hddsDatanode is not defined kms_1 | class org.apache.hadoop.crypto.key.kms.server.KMSJSONReader scm_1 | 2019-10-07 08:08:30 INFO Server:1074 - Starting Socket Reader #1 for port 9861 datanode_2 | 2019-10-07 08:08:33 INFO log:192 - Logging initialized @7719ms om_1 | ************************************************************/ datanode_3 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.w.WebAppContext@3cdff901{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-3298991593031954848.dir/webapp/,AVAILABLE}{/hddsDatanode} s3g_1 | 2019-10-07 08:15:46 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 datanode_1 | 2019-10-07 08:08:33 INFO HttpServer2:975 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) kms_1 | Oct 07, 2019 8:08:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate datanode_2 | 2019-10-07 08:08:33 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. om_1 | 2019-10-07 08:08:34 INFO OzoneManagerStarter:51 - registered UNIX signal handlers for [TERM, HUP, INT] datanode_3 | 2019-10-07 08:08:33 INFO AbstractConnector:278 - Started ServerConnector@285005b8{HTTP/1.1,[http/1.1]}{0.0.0.0:9882} scm_1 | 2019-10-07 08:08:30 INFO CallQueueManager:84 - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. s3g_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_1 | 2019-10-07 08:08:33 INFO HttpServer2:948 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context hddsDatanode kms_1 | INFO: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' datanode_2 | 2019-10-07 08:08:33 INFO HttpRequestLog:81 - Http request log for http.requests.hddsDatanode is not defined om_1 | 2019-10-07 08:08:35 INFO OMHANodeDetails:190 - Configuration either no ozone.om.address set. Falling back to the default OM address om/172.18.0.2:9862 datanode_3 | 2019-10-07 08:08:33 INFO Server:419 - Started @7300ms scm_1 | 2019-10-07 08:08:30 INFO Server:1074 - Starting Socket Reader #1 for port 9863 s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:33 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs datanode_2 | 2019-10-07 08:08:33 INFO HttpServer2:975 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) om_1 | 2019-10-07 08:08:35 INFO OMHANodeDetails:220 - OM Service ID is not set. Setting it to the default ID: omServiceIdDefault scm_1 | 2019-10-07 08:08:30 INFO CallQueueManager:84 - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. datanode_3 | 2019-10-07 08:08:33 INFO MetricsSinkAdapter:204 - Sink prometheus started s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_1 | 2019-10-07 08:08:33 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:33 INFO HttpServer2:948 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context hddsDatanode om_1 | 2019-10-07 08:08:35 WARN ServerUtils:222 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. scm_1 | 2019-10-07 08:08:30 INFO Server:1074 - Starting Socket Reader #1 for port 9860 datanode_3 | 2019-10-07 08:08:33 INFO MetricsSystemImpl:301 - Registered sink prometheus s3g_1 | 2019-10-07 08:15:46 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_1 | 2019-10-07 08:08:33 DEBUG HttpServer2:857 - Found existing conf servlet at path /conf; will replace mapping with conf servlet kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:33 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs om_1 | WARNING: An illegal reflective access operation has occurred scm_1 | 2019-10-07 08:08:31 INFO DFSUtil:1629 - Starting web server as: HTTP/scm@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:33 INFO BaseHttpServer:215 - HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882 s3g_1 | 20191007T081546Z datanode_1 | 2019-10-07 08:08:33 DEBUG HttpServer2:1273 - opening listeners: [ServerConnector@27585351{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}] kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_2 | 2019-10-07 08:08:33 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static om_1 | WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar) to method sun.security.krb5.Config.getInstance() scm_1 | 2019-10-07 08:08:31 INFO DFSUtil:1641 - Starting Web-server for scm at: http://0.0.0.0:9876 s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:33 INFO JvmPauseMonitor:188 - Starting JVM pause monitor datanode_1 | 2019-10-07 08:08:33 INFO HttpServer2:1191 - Jetty bound to port 9882 datanode_2 | 2019-10-07 08:08:33 DEBUG HttpServer2:857 - Found existing conf servlet at path /conf; will replace mapping with conf servlet om_1 | WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:08:31 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 2 failover attempts. Trying to failover immediately. datanode_3 | 2019-10-07 08:08:35 INFO OzoneContainer:213 - Attempting to start container services. datanode_2 | 2019-10-07 08:08:33 DEBUG HttpServer2:1273 - opening listeners: [ServerConnector@27585351{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}] datanode_1 | 2019-10-07 08:08:33 INFO Server:351 - jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 om_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:08:31 INFO HttpRequestLog:81 - Http request log for http.requests.scm is not defined datanode_3 | 2019-10-07 08:08:35 INFO OzoneContainer:177 - Background container scanner has been disabled. datanode_2 | 2019-10-07 08:08:33 INFO HttpServer2:1191 - Jetty bound to port 9882 datanode_1 | 2019-10-07 08:08:33 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. om_1 | WARNING: All illegal access operations will be denied in a future release scm_1 | 2019-10-07 08:08:31 INFO HttpServer2:975 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) datanode_3 | 2019-10-07 08:08:35 INFO XceiverServerRatis:418 - Starting XceiverServerRatis 8b31d836-555f-487d-9990-0cc47031ea16 at port 9858 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 s3g_1 | 2019-10-07 08:15:46 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_2 | 2019-10-07 08:08:33 INFO Server:351 - jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_1 | 2019-10-07 08:08:33 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/ad18a6fa7980@EXAMPLE.COM om_1 | 2019-10-07 08:08:35 INFO UserGroupInformation:1009 - Login successful for user om/om@EXAMPLE.COM using keytab file /etc/security/keytabs/om.keytab datanode_3 | 2019-10-07 08:08:35 INFO RaftServerProxy:299 - 8b31d836-555f-487d-9990-0cc47031ea16: start RPC server scm_1 | 2019-10-07 08:08:31 INFO HttpServer2:948 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context scm kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_1 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@5fdfe8cf{/logs,file:///var/log/hadoop/,AVAILABLE} s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:33 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_3 | 2019-10-07 08:08:35 INFO GrpcService:158 - 8b31d836-555f-487d-9990-0cc47031ea16: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858 scm_1 | 2019-10-07 08:08:31 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static datanode_1 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@31723307{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:08:35 INFO OzoneManager:752 - Ozone Manager login successful. datanode_2 | 2019-10-07 08:08:33 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/311de0fd64d4@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:38 INFO RaftServerProxy:89 - 8b31d836-555f-487d-9990-0cc47031ea16: addNew group-8345266181FE:[8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858] returns group-8345266181FE:java.util.concurrent.CompletableFuture@692ab8d8[Not completed] scm_1 | 2019-10-07 08:08:31 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs datanode_1 | 2019-10-07 08:08:33 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/ad18a6fa7980@EXAMPLE.COM kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:08:35 WARN ServerUtils:222 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_2 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@5fdfe8cf{/logs,file:///var/log/hadoop/,AVAILABLE} datanode_1 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.w.WebAppContext@2f054f70{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-40832158342884333.dir/webapp/,AVAILABLE}{/hddsDatanode} s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:38 INFO RaftServerImpl:95 - 8b31d836-555f-487d-9990-0cc47031ea16: new RaftServerImpl for group-8345266181FE:[8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858] with ContainerStateMachine:uninitialized om_1 | 2019-10-07 08:08:36 INFO OMCertificateClient:118 - Loading certificate from location:/data/metadata/om/certs. datanode_1 | 2019-10-07 08:08:33 INFO AbstractConnector:278 - Started ServerConnector@27585351{HTTP/1.1,[http/1.1]}{0.0.0.0:9882} scm_1 | 2019-10-07 08:08:31 DEBUG HttpServer2:857 - Found existing conf servlet at path /conf; will replace mapping with conf servlet s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_2 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@31723307{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.min = 5s (custom) om_1 | 2019-10-07 08:08:36 INFO OMCertificateClient:148 - Added certificate from file:/data/metadata/om/certs/1099937157938208.crt. datanode_1 | 2019-10-07 08:08:33 INFO Server:419 - Started @7363ms scm_1 | 2019-10-07 08:08:31 INFO StorageContainerManager:770 - StorageContainerLocationProtocol RPC server is listening at /0.0.0.0:9860 s3g_1 | 2019-10-07 08:15:46 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_2 | 2019-10-07 08:08:33 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/311de0fd64d4@EXAMPLE.COM kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.max = 5200ms (custom) om_1 | 2019-10-07 08:08:36 INFO OMCertificateClient:148 - Added certificate from file:/data/metadata/om/certs/CA-1.crt. datanode_1 | 2019-10-07 08:08:33 INFO MetricsSinkAdapter:204 - Sink prometheus started scm_1 | 2019-10-07 08:08:31 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:33 INFO ContextHandler:781 - Started o.e.j.w.WebAppContext@2f054f70{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-17656943389114920161.dir/webapp/,AVAILABLE}{/hddsDatanode} kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.rpcslowness.timeout = 120s (custom) om_1 | 2019-10-07 08:08:36 WARN ServerUtils:222 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_1 | 2019-10-07 08:08:33 INFO MetricsSystemImpl:301 - Registered sink prometheus scm_1 | 2019-10-07 08:08:31 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_2 | 2019-10-07 08:08:33 INFO AbstractConnector:278 - Started ServerConnector@27585351{HTTP/1.1,[http/1.1]}{0.0.0.0:9882} datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.sleep.deviation.threshold = 300 (default) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:33 INFO BaseHttpServer:215 - HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882 om_1 | 2019-10-07 08:08:36 INFO log:192 - Logging initialized @2779ms scm_1 | 2019-10-07 08:08:31 INFO MetricsSystemImpl:191 - StorageContainerManager metrics system started datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 3 failover attempts. Trying to failover immediately. datanode_1 | 2019-10-07 08:08:33 INFO JvmPauseMonitor:188 - Starting JVM pause monitor om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: userTable datanode_3 | 2019-10-07 08:08:38 INFO RaftServerImpl:103 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: ConfigurationManager, init=-1: [8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858], old=null, confs= scm_1 | 2019-10-07 08:08:31 INFO SCMClientProtocolServer:159 - RPC server for Client is listening at /0.0.0.0:9860 datanode_2 | 2019-10-07 08:08:33 INFO Server:419 - Started @7947ms kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 2019-10-07 08:15:46 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_1 | 2019-10-07 08:08:35 INFO OzoneContainer:213 - Attempting to start container services. om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:userTable datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) scm_1 | 2019-10-07 08:08:31 INFO Server:1314 - IPC Server Responder: starting datanode_2 | 2019-10-07 08:08:33 INFO MetricsSinkAdapter:204 - Sink prometheus started kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007T081546Z datanode_1 | 2019-10-07 08:08:35 INFO OzoneContainer:177 - Background container scanner has been disabled. om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: volumeTable datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.corruption.policy = EXCEPTION (default) datanode_2 | 2019-10-07 08:08:33 INFO MetricsSystemImpl:301 - Registered sink prometheus scm_1 | 2019-10-07 08:08:31 INFO Server:1153 - IPC Server listener on 9860: starting kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_1 | 2019-10-07 08:08:35 INFO XceiverServerRatis:418 - Starting XceiverServerRatis 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765 at port 9858 om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:volumeTable datanode_3 | 2019-10-07 08:08:38 INFO RaftStorageDirectory:246 - The storage directory /data/metadata/ratis/b8de9a03-0aa9-4b04-82bb-8345266181fe does not exist. Creating ... datanode_2 | 2019-10-07 08:08:33 INFO BaseHttpServer:215 - HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882 scm_1 | 2019-10-07 08:08:31 INFO StorageContainerManager:780 - ScmBlockLocationProtocol RPC server is listening at /0.0.0.0:9863 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_1 | 2019-10-07 08:08:35 INFO RaftServerProxy:299 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start RPC server om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: bucketTable datanode_2 | 2019-10-07 08:08:33 INFO JvmPauseMonitor:188 - Starting JVM pause monitor datanode_3 | 2019-10-07 08:08:38 INFO RaftStorageDirectory:328 - Lock on /data/metadata/ratis/b8de9a03-0aa9-4b04-82bb-8345266181fe/in_use.lock acquired by nodename 7@d8b9c0e14ce8 scm_1 | 2019-10-07 08:08:31 INFO SCMBlockProtocolServer:146 - RPC server for Block Protocol is listening at /0.0.0.0:9863 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 2019-10-07 08:15:46 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_1 | 2019-10-07 08:08:35 INFO GrpcService:158 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858 om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:bucketTable datanode_2 | 2019-10-07 08:08:35 INFO OzoneContainer:213 - Attempting to start container services. datanode_3 | 2019-10-07 08:08:38 INFO RaftStorage:84 - Storage directory /data/metadata/ratis/b8de9a03-0aa9-4b04-82bb-8345266181fe has been successfully formatted. scm_1 | 2019-10-07 08:08:31 INFO Server:1314 - IPC Server Responder: starting kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:35 INFO OzoneContainer:177 - Background container scanner has been disabled. om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: keyTable datanode_1 | 2019-10-07 08:08:39 INFO RaftServerProxy:89 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: addNew group-D6EEC858101A:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858] returns group-D6EEC858101A:java.util.concurrent.CompletableFuture@20c8ac74[Not completed] datanode_3 | 2019-10-07 08:08:38 INFO ContainerStateMachine:231 - group-8345266181FE: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) scm_1 | 2019-10-07 08:08:31 INFO Server:1153 - IPC Server listener on 9863: starting kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_2 | 2019-10-07 08:08:35 INFO XceiverServerRatis:418 - Starting XceiverServerRatis 27f27e32-1f1d-417d-8e7c-554581f7cfc6 at port 9858 om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:keyTable datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.notification.no-leader.timeout = 120s (custom) scm_1 | 2019-10-07 08:08:31 INFO StorageContainerManager:784 - ScmDatanodeProtocl RPC server is listening at /0.0.0.0:9861 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 4 failover attempts. Trying to failover immediately. datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:95 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: new RaftServerImpl for group-D6EEC858101A:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858] with ContainerStateMachine:uninitialized om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: deletedTable datanode_2 | 2019-10-07 08:08:35 INFO RaftServerProxy:299 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start RPC server datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.use.memory = false (default) scm_1 | 2019-10-07 08:08:31 INFO SCMDatanodeProtocolServer:169 - RPC server for DataNodes is listening at /0.0.0.0:9861 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.min = 5s (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:deletedTable datanode_2 | 2019-10-07 08:08:35 INFO GrpcService:158 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858 datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.purge.gap = 1000000 (custom) scm_1 | 2019-10-07 08:08:31 INFO Server:1314 - IPC Server Responder: starting kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007T081546Z datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.max = 5200ms (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: openKeyTable datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) scm_1 | 2019-10-07 08:08:31 INFO Server:1153 - IPC Server listener on 9861: starting kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerProxy:89 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: addNew group-7BA479E1AFB4:[27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] returns group-7BA479E1AFB4:java.util.concurrent.CompletableFuture@5481a2ba[Not completed] s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpcslowness.timeout = 120s (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:openKeyTable datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerImpl:95 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: new RaftServerImpl for group-7BA479E1AFB4:[27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] with ContainerStateMachine:uninitialized s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.sleep.deviation.threshold = 300 (default) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: s3Table scm_1 | 2019-10-07 08:08:31 INFO SCMClientProtocolServer:204 - Starting RPC server for SCMSecurityProtocolServer. is listening at /0.0.0.0:9961 datanode_3 | 2019-10-07 08:08:38 WARN MetricsSystemImpl:151 - HddsDatanode metrics system already initialized! kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.min = 5s (custom) s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:s3Table datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) scm_1 | 2019-10-07 08:08:31 INFO Server:1314 - IPC Server Responder: starting datanode_3 | 2019-10-07 08:08:38 INFO MetricRegistries:64 - Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.max = 5200ms (custom) s3g_1 | 20191007T081546Z om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: multipartInfoTable datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:103 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: ConfigurationManager, init=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858], old=null, confs= scm_1 | 2019-10-07 08:08:31 INFO Server:1153 - IPC Server listener on 9961: starting datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.segment.cache.num.max = 2 (custom) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.rpcslowness.timeout = 120s (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:multipartInfoTable datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_3 | 2019-10-07 08:08:38 INFO SegmentedRaftLogWorker:173 - new 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/b8de9a03-0aa9-4b04-82bb-8345266181fe scm_1 | 2019-10-07 08:08:31 DEBUG HttpServer2:1273 - opening listeners: [ServerConnector@4bc59b27{HTTP/1.1,[http/1.1]}{0.0.0.0:9876}] s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.sleep.deviation.threshold = 300 (default) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: dTokenTable datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.corruption.policy = EXCEPTION (default) datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.queue.byte-limit = 2147483647 (custom) scm_1 | 2019-10-07 08:08:31 INFO HttpServer2:1191 - Jetty bound to port 9876 s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 5 failover attempts. Trying to failover immediately. kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:dTokenTable datanode_1 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:246 - The storage directory /data/metadata/ratis/8431418d-9c49-4a90-bf6b-d6eec858101a does not exist. Creating ... scm_1 | 2019-10-07 08:08:31 INFO Server:351 - jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerImpl:103 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: ConfigurationManager, init=-1: [27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null, confs= datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.queue.element-limit = 1024 (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: s3SecretTable datanode_1 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:328 - Lock on /data/metadata/ratis/8431418d-9c49-4a90-bf6b-d6eec858101a/in_use.lock acquired by nodename 7@ad18a6fa7980 scm_1 | 2019-10-07 08:08:31 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:s3SecretTable datanode_1 | 2019-10-07 08:08:39 INFO RaftStorage:84 - Storage directory /data/metadata/ratis/8431418d-9c49-4a90-bf6b-d6eec858101a has been successfully formatted. scm_1 | 2019-10-07 08:08:31 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/scm@EXAMPLE.COM s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.corruption.policy = EXCEPTION (default) datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.preallocated.size = 16384 (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: prefixTable datanode_1 | 2019-10-07 08:08:39 INFO ContainerStateMachine:231 - group-D6EEC858101A: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) scm_1 | 2019-10-07 08:08:31 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@3003827c{/logs,file:///var/log/hadoop/,AVAILABLE} s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_2 | 2019-10-07 08:08:38 INFO RaftStorageDirectory:246 - The storage directory /data/metadata/ratis/65453df7-dd08-4275-9d28-7ba479e1afb4 does not exist. Creating ... datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.write.buffer.size = 33554432 (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:168 - Using default column profile:DBProfile.DISK for Table:prefixTable datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.notification.no-leader.timeout = 120s (custom) scm_1 | 2019-10-07 08:08:31 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@513b52af{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:21 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:38 INFO RaftStorageDirectory:328 - Lock on /data/metadata/ratis/65453df7-dd08-4275-9d28-7ba479e1afb4/in_use.lock acquired by nodename 7@311de0fd64d4 datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.force.sync.num = 128 (default) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.use.memory = false (default) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:117 - using custom profile for table: default scm_1 | 2019-10-07 08:08:31 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/scm@EXAMPLE.COM s3g_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_2 | 2019-10-07 08:08:38 INFO RaftStorage:84 - Storage directory /data/metadata/ratis/65453df7-dd08-4275-9d28-7ba479e1afb4 has been successfully formatted. datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync = true (default) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.purge.gap = 1000000 (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:174 - Using default column profile:DBProfile.DISK for Table:default scm_1 | 2019-10-07 08:08:31 INFO ContextHandler:781 - Started o.e.j.w.WebAppContext@1e3df614{/,file:///tmp/jetty-0.0.0.0-9876-scm-_-any-1562936238714719883.dir/webapp/,AVAILABLE}{/scm} s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_2 | 2019-10-07 08:08:38 INFO ContainerStateMachine:231 - group-7BA479E1AFB4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) kdc_1 | Oct 07 08:08:21 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) om_1 | 2019-10-07 08:08:36 INFO DBStoreBuilder:205 - Using default options. DBProfile.DISK scm_1 | 2019-10-07 08:08:31 INFO AbstractConnector:278 - Started ServerConnector@4bc59b27{HTTP/1.1,[http/1.1]}{0.0.0.0:9876} datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout = 10s (default) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 6 failover attempts. Trying to failover immediately. datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.notification.no-leader.timeout = 120s (custom) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) om_1 | 2019-10-07 08:08:36 INFO deprecation:1394 - No unit for ozone.manager.delegation.remover.scan.interval(3600000) assuming MILLISECONDS scm_1 | 2019-10-07 08:08:31 INFO Server:419 - Started @4339ms datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.use.memory = false (default) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 datanode_1 | 2019-10-07 08:08:39 WARN MetricsSystemImpl:151 - HddsDatanode metrics system already initialized! om_1 | 2019-10-07 08:08:36 INFO OzoneSecretStore:69 - Loaded 0 tokens scm_1 | 2019-10-07 08:08:31 INFO MetricsSinkAdapter:204 - Sink prometheus started datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.caching.enabled = true (custom) s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.purge.gap = 1000000 (custom) kdc_1 | Entry for principal dn/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.s3g.keytab. datanode_1 | 2019-10-07 08:08:39 INFO MetricRegistries:64 - Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl om_1 | 2019-10-07 08:08:36 INFO OzoneDelegationTokenSecretManager:424 - Loading token state into token manager. scm_1 | 2019-10-07 08:08:31 INFO MetricsSystemImpl:301 - Registered sink prometheus datanode_3 | 2019-10-07 08:08:38 INFO SegmentedRaftLogWorker:128 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) kdc_1 | Entry for principal dn/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.s3g.keytab. datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.cache.num.max = 2 (custom) om_1 | 2019-10-07 08:08:36 INFO CallQueueManager:84 - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 2000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. scm_1 | 2019-10-07 08:08:31 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.enabled = true (custom) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_2 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) kdc_1 | Generiting keytab datanode_1 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:173 - new 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/8431418d-9c49-4a90-bf6b-d6eec858101a om_1 | 2019-10-07 08:08:36 INFO Server:1074 - Starting Socket Reader #1 for port 9862 scm_1 | 2019-10-07 08:08:31 INFO BaseHttpServer:215 - HTTP server of SCM is listening at http://0.0.0.0:9876 datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.threshold = 10000 (custom) s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_2 | 2019-10-07 08:08:38 WARN MetricsSystemImpl:151 - HddsDatanode metrics system already initialized! kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.byte-limit = 2147483647 (custom) om_1 | 2019-10-07 08:08:36 INFO OzoneManager:1069 - OzoneManager RPC server is listening at om/172.18.0.2:9862 scm_1 | 2019-10-07 08:08:31 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.snapshot.retention.file.num = 5 (custom) s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:39 INFO MetricRegistries:64 - Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl kdc_1 | WARNING: no policy specified for om/scm@EXAMPLE.COM; defaulting to no policy om_1 | 2019-10-07 08:08:36 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.element-limit = 1024 (custom) scm_1 | 2019-10-07 08:08:31 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:38 INFO RaftServerConfigKeys:43 - raft.server.retrycache.expirytime = 600000ms (custom) s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.cache.num.max = 2 (custom) kdc_1 | Principal "om/scm@EXAMPLE.COM" created. om_1 | 2019-10-07 08:08:37 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). scm_1 | 2019-10-07 08:08:31 INFO JvmPauseMonitor:188 - Starting JVM pause monitor datanode_3 | 2019-10-07 08:08:38 INFO RaftServerImpl:181 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: start as a follower, conf=-1: [8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858], old=null s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 7 failover attempts. Trying to failover immediately. datanode_2 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:173 - new 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/65453df7-dd08-4275-9d28-7ba479e1afb4 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:08:37 INFO MetricsSystemImpl:191 - OzoneManager metrics system started datanode_3 | 2019-10-07 08:08:38 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.byte-limit = 2147483647 (custom) scm_1 | 2019-10-07 08:08:31 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:08:37 INFO OzoneManager:713 - Reading keypair and certificate from file system. datanode_3 | 2019-10-07 08:08:38 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start FollowerState datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.element-limit = 1024 (custom) scm_1 | 2019-10-07 08:08:31 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.preallocated.size = 16384 (custom) s3g_1 | 20191007T081546Z om_1 | 2019-10-07 08:08:37 INFO OzoneManager:679 - Starting OM block token secret manager datanode_3 | 2019-10-07 08:08:38 INFO JmxRegister:44 - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-8345266181FE,id=8b31d836-555f-487d-9990-0cc47031ea16 datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) scm_1 | 2019-10-07 08:08:31 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.write.buffer.size = 33554432 (custom) om_1 | 2019-10-07 08:08:37 INFO OzoneBlockTokenSecretManager:168 - Updating the current master key for generating tokens s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:39 INFO RaftServerProxy:89 - 8b31d836-555f-487d-9990-0cc47031ea16: addNew group-A621526D5DE4:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] returns group-A621526D5DE4:java.util.concurrent.CompletableFuture@5c1772a[Not completed] kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.preallocated.size = 16384 (custom) scm_1 | 2019-10-07 08:08:31 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.force.sync.num = 128 (default) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_3 | 2019-10-07 08:08:39 INFO RaftServerImpl:95 - 8b31d836-555f-487d-9990-0cc47031ea16: new RaftServerImpl for group-A621526D5DE4:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] with ContainerStateMachine:uninitialized kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.write.buffer.size = 33554432 (custom) scm_1 | 2019-10-07 08:08:31 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol om_1 | 2019-10-07 08:08:37 INFO OzoneManager:690 - Starting OM delegation token secret manager datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync = true (default) s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.min = 5s (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.force.sync.num = 128 (default) scm_1 | 2019-10-07 08:08:31 INFO SCMClientProtocolServer:114 - Processing CSR for dn 311de0fd64d4, UUID: 27f27e32-1f1d-417d-8e7c-554581f7cfc6 om_1 | 2019-10-07 08:08:37 INFO OzoneDelegationTokenSecretManager:168 - Updating the current master key for generating tokens datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.max = 5200ms (custom) s3g_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync = true (default) scm_1 | 2019-10-07 08:08:31 INFO SCMClientProtocolServer:114 - Processing CSR for dn ad18a6fa7980, UUID: 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765 om_1 | 2019-10-07 08:08:37 INFO OzoneDelegationTokenSecretManager:529 - Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpcslowness.timeout = 120s (custom) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout = 10s (default) scm_1 | 2019-10-07 08:08:32 INFO SCMClientProtocolServer:114 - Processing CSR for dn d8b9c0e14ce8, UUID: 8b31d836-555f-487d-9990-0cc47031ea16 om_1 | 2019-10-07 08:08:37 INFO Server:1314 - IPC Server Responder: starting datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.caching.enabled = true (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 8 failover attempts. Trying to failover immediately. datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.sleep.deviation.threshold = 300 (default) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) scm_1 | 2019-10-07 08:08:33 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:37 INFO Server:1153 - IPC Server listener on 9862: starting datanode_1 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:128 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.caching.enabled = true (custom) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 scm_1 | 2019-10-07 08:08:33 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol om_1 | 2019-10-07 08:08:37 INFO DFSUtil:1629 - Starting web server as: HTTP/om@EXAMPLE.COM datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.enabled = true (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_3 | 2019-10-07 08:08:39 INFO RaftServerImpl:103 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: ConfigurationManager, init=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null, confs= datanode_2 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:128 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 s3g_1 | 20191007T081546Z scm_1 | 2019-10-07 08:08:33 INFO SCMClientProtocolServer:139 - Processing CSR for om om, UUID: 5c322c54-10fd-4e62-a53e-8da8fa9d92e8 om_1 | 2019-10-07 08:08:37 INFO DFSUtil:1641 - Starting Web-server for ozoneManager at: http://0.0.0.0:9874 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.threshold = 10000 (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.enabled = true (custom) s3g_1 | 20191007/us-west-1/s3/aws4_request scm_1 | 2019-10-07 08:08:35 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:37 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.retention.file.num = 5 (custom) datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.threshold = 10000 (custom) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf scm_1 | 2019-10-07 08:08:35 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:37 INFO HttpRequestLog:81 - Http request log for http.requests.ozoneManager is not defined datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.corruption.policy = EXCEPTION (default) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.retention.file.num = 5 (custom) s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:08:37 INFO HttpServer2:975 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) scm_1 | 2019-10-07 08:08:35 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:246 - The storage directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 does not exist. Creating ... kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:181 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: start as a follower, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858], old=null datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.retrycache.expirytime = 600000ms (custom) s3g_1 | 20191007T081546Z scm_1 | 2019-10-07 08:08:35 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:328 - Lock on /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4/in_use.lock acquired by nodename 7@d8b9c0e14ce8 om_1 | 2019-10-07 08:08:37 INFO HttpServer2:948 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context ozoneManager kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | 2019-10-07 08:08:39 INFO RaftServerImpl:181 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: start as a follower, conf=-1: [27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null s3g_1 | 20191007/us-west-1/s3/aws4_request scm_1 | 2019-10-07 08:08:35 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:39 INFO RaftStorage:84 - Storage directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 has been successfully formatted. om_1 | 2019-10-07 08:08:37 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_1 | 2019-10-07 08:08:39 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start FollowerState datanode_2 | 2019-10-07 08:08:39 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: changes role from null to FOLLOWER at term 0 for startAsFollower s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 9 failover attempts. Trying to failover immediately. scm_1 | 2019-10-07 08:08:35 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:39 INFO ContainerStateMachine:231 - group-A621526D5DE4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) om_1 | 2019-10-07 08:08:37 INFO HttpServer2:958 - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start FollowerState datanode_1 | 2019-10-07 08:08:39 INFO JmxRegister:44 - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-D6EEC858101A,id=8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765 s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.notification.no-leader.timeout = 120s (custom) scm_1 | 2019-10-07 08:08:35 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:37 DEBUG HttpServer2:857 - Found existing conf servlet at path /conf; will replace mapping with conf servlet kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_2 | 2019-10-07 08:08:39 INFO JmxRegister:44 - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-7BA479E1AFB4,id=27f27e32-1f1d-417d-8e7c-554581f7cfc6 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerProxy:89 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: addNew group-A621526D5DE4:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] returns group-A621526D5DE4:java.util.concurrent.CompletableFuture@75f09e75[Not completed] s3g_1 | 20191007T081546Z datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.use.memory = false (default) scm_1 | 2019-10-07 08:08:35 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol om_1 | 2019-10-07 08:08:37 DEBUG HttpServer2:1273 - opening listeners: [ServerConnector@2ba1209b{HTTP/1.1,[http/1.1]}{0.0.0.0:9874}] kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerProxy:89 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: addNew group-A621526D5DE4:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] returns group-A621526D5DE4:java.util.concurrent.CompletableFuture@1a21399d[Not completed] datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:95 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: new RaftServerImpl for group-A621526D5DE4:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] with ContainerStateMachine:uninitialized s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.purge.gap = 1000000 (custom) scm_1 | 2019-10-07 08:08:37 INFO NetworkTopology:111 - Added a new node: /default-rack/8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765 om_1 | 2019-10-07 08:08:37 INFO HttpServer2:1191 - Jetty bound to port 9874 kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database datanode_2 | 2019-10-07 08:08:39 INFO RaftServerImpl:95 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: new RaftServerImpl for group-A621526D5DE4:[8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858] with ContainerStateMachine:uninitialized datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.min = 5s (custom) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) scm_1 | 2019-10-07 08:08:37 INFO NetworkTopology:111 - Added a new node: /default-rack/8b31d836-555f-487d-9990-0cc47031ea16 om_1 | 2019-10-07 08:08:37 INFO Server:351 - jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.min = 5s (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.max = 5200ms (custom) s3g_1 | 2019-10-07 08:15:47 ERROR OMFailoverProxyProvider:268 - Failed to connect to OM. Attempted 10 retries and 10 failovers datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) kdc_1 | Oct 07 08:08:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435701, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:08:37 INFO SCMNodeManager:268 - Registered Data node : 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765{ip: 172.18.0.8, host: ozonesecure_datanode_1.ozonesecure_default, networkLocation: /default-rack, certSerialId: 1099935960379922} om_1 | 2019-10-07 08:08:37 INFO AuthenticationFilter:240 - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpc.timeout.max = 5200ms (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpcslowness.timeout = 120s (custom) s3g_1 | 2019-10-07 08:15:47 ERROR OzoneClientFactory:259 - Couldn't create RpcClient protocol exception: datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.cache.num.max = 2 (custom) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:08:37 INFO SCMNodeManager:268 - Registered Data node : 8b31d836-555f-487d-9990-0cc47031ea16{ip: 172.18.0.10, host: ozonesecure_datanode_3.ozonesecure_default, networkLocation: /default-rack, certSerialId: 1099936009160445} om_1 | 2019-10-07 08:08:37 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/om@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.rpcslowness.timeout = 120s (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.sleep.deviation.threshold = 300 (default) datanode_3 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:173 - new 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 s3g_1 | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:08:37 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@42507640{/logs,file:///var/log/hadoop/,AVAILABLE} scm_1 | 2019-10-07 08:08:37 INFO SCMSafeModeManager:71 - SCM in safe mode. 1 DataNodes registered, 1 required. datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.sleep.deviation.threshold = 300 (default) datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:08:37 INFO ContextHandler:781 - Started o.e.j.s.ServletContextHandler@12f279b5{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} scm_1 | 2019-10-07 08:08:37 INFO SCMSafeModeManager:177 - ScmSafeModeManager, all rules are successfully validated s3g_1 | 20191007T081546Z datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:103 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: ConfigurationManager, init=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null, confs= kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:08:37 INFO KerberosAuthenticationHandler:164 - Using keytab /etc/security/keytabs/HTTP.keytab, for principal HTTP/om@EXAMPLE.COM scm_1 | 2019-10-07 08:08:37 INFO SCMSafeModeManager:193 - SCM exiting safe mode. s3g_1 | 20191007/us-west-1/s3/aws4_request datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerImpl:103 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: ConfigurationManager, init=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null, confs= datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:08:37 INFO ContextHandler:781 - Started o.e.j.w.WebAppContext@2eda4eeb{/,file:///tmp/jetty-0.0.0.0-9874-ozoneManager-_-any-4060432635598829009.dir/webapp/,AVAILABLE}{/ozoneManager} scm_1 | 2019-10-07 08:08:37 INFO NetworkTopology:111 - Added a new node: /default-rack/27f27e32-1f1d-417d-8e7c-554581f7cfc6 datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.corruption.policy = EXCEPTION (default) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:08:37 INFO AbstractConnector:278 - Started ServerConnector@2ba1209b{HTTP/1.1,[http/1.1]}{0.0.0.0:9874} scm_1 | 2019-10-07 08:08:37 INFO SCMNodeManager:268 - Registered Data node : 27f27e32-1f1d-417d-8e7c-554581f7cfc6{ip: 172.18.0.5, host: ozonesecure_datanode_2.ozonesecure_default, networkLocation: /default-rack, certSerialId: 1099935960397927} datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.preallocated.size = 16384 (custom) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.corruption.policy = EXCEPTION (default) datanode_1 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:246 - The storage directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 does not exist. Creating ... om_1 | 2019-10-07 08:08:37 INFO Server:419 - Started @4030ms kdc_1 | Entry for principal om/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.scm.keytab. scm_1 | 2019-10-07 08:08:38 INFO PipelineStateManager:56 - Created pipeline Pipeline[ Id: b8de9a03-0aa9-4b04-82bb-8345266181fe, Nodes: 8b31d836-555f-487d-9990-0cc47031ea16{ip: 172.18.0.10, host: ozonesecure_datanode_3.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.write.buffer.size = 33554432 (custom) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) datanode_2 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:246 - The storage directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 does not exist. Creating ... datanode_1 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:328 - Lock on /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4/in_use.lock acquired by nodename 7@ad18a6fa7980 om_1 | 2019-10-07 08:08:37 INFO MetricsSinkAdapter:204 - Sink prometheus started kdc_1 | Entry for principal om/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.scm.keytab. scm_1 | 2019-10-07 08:08:39 INFO PipelineStateManager:56 - Created pipeline Pipeline[ Id: 65453df7-dd08-4275-9d28-7ba479e1afb4, Nodes: 27f27e32-1f1d-417d-8e7c-554581f7cfc6{ip: 172.18.0.5, host: ozonesecure_datanode_2.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.force.sync.num = 128 (default) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) datanode_2 | 2019-10-07 08:08:39 INFO RaftStorageDirectory:328 - Lock on /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4/in_use.lock acquired by nodename 7@311de0fd64d4 om_1 | 2019-10-07 08:08:37 INFO MetricsSystemImpl:301 - Registered sink prometheus kdc_1 | Generiting keytab s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync = true (default) datanode_1 | 2019-10-07 08:08:39 INFO RaftStorage:84 - Storage directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 has been successfully formatted. datanode_2 | 2019-10-07 08:08:39 INFO RaftStorage:84 - Storage directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 has been successfully formatted. scm_1 | 2019-10-07 08:08:39 INFO PipelineStateManager:56 - Created pipeline Pipeline[ Id: 8431418d-9c49-4a90-bf6b-d6eec858101a, Nodes: 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765{ip: 172.18.0.8, host: ozonesecure_datanode_1.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] om_1 | 2019-10-07 08:08:37 INFO BaseHttpServer:215 - HTTP server of OZONEMANAGER is listening at http://0.0.0.0:9874 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_1 | 2019-10-07 08:08:39 INFO ContainerStateMachine:231 - group-A621526D5DE4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_2 | 2019-10-07 08:08:39 INFO ContainerStateMachine:231 - group-A621526D5DE4: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) scm_1 | 2019-10-07 08:08:39 INFO PipelineStateManager:56 - Created pipeline Pipeline[ Id: c0883aad-232c-4e37-8411-a621526d5de4, Nodes: 8b31d836-555f-487d-9990-0cc47031ea16{ip: 172.18.0.10, host: ozonesecure_datanode_3.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}27f27e32-1f1d-417d-8e7c-554581f7cfc6{ip: 172.18.0.5, host: ozonesecure_datanode_2.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765{ip: 172.18.0.8, host: ozonesecure_datanode_1.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] om_1 | 2019-10-07 08:08:53 INFO Server:1849 - Auth successful for HTTP/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | WARNING: no policy specified for scm/om@EXAMPLE.COM; defaulting to no policy datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.notification.no-leader.timeout = 120s (custom) scm_1 | 2019-10-07 08:08:54 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:53 INFO ServiceAuthorizationManager:138 - Authorization successful for HTTP/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) kdc_1 | Principal "scm/om@EXAMPLE.COM" created. datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.use.memory = false (default) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.use.memory = false (default) scm_1 | 2019-10-07 08:08:54 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol om_1 | 2019-10-07 08:08:55 INFO OMVolumeCreateRequest:193 - created volume:vol-0-99606 for user:HTTP/scm@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_3 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:128 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.purge.gap = 1000000 (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.purge.gap = 1000000 (custom) scm_1 | 2019-10-07 08:08:55 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:55 INFO OMVolumeCreateRequest:193 - created volume:vol-1-04353 for user:HTTP/scm@EXAMPLE.COM s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Entry for principal scm/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.om.keytab. datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) scm_1 | 2019-10-07 08:08:55 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol om_1 | 2019-10-07 08:08:55 INFO OMVolumeCreateRequest:193 - created volume:vol-2-55464 for user:HTTP/scm@EXAMPLE.COM s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) kdc_1 | Entry for principal scm/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.om.keytab. datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) scm_1 | 2019-10-07 08:08:55 INFO PipelineStateManager:56 - Created pipeline Pipeline[ Id: 3e5018c3-d991-40a0-bc8e-302129549cdd, Nodes: 27f27e32-1f1d-417d-8e7c-554581f7cfc6{ip: 172.18.0.5, host: ozonesecure_datanode_2.ozonesecure_default, networkLocation: /default-rack, certSerialId: null}, Type:STAND_ALONE, Factor:ONE, State:OPEN] om_1 | 2019-10-07 08:08:55 INFO OMVolumeCreateRequest:193 - created volume:vol-3-02689 for user:HTTP/scm@EXAMPLE.COM s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) kdc_1 | Generiting keytab datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.retention.file.num = 5 (custom) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.cache.num.max = 2 (custom) scm_1 | 2019-10-07 08:08:56 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:08:55 INFO OMVolumeCreateRequest:193 - created volume:vol-4-41040 for user:HTTP/scm@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.cache.num.max = 2 (custom) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_3 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.retrycache.expirytime = 600000ms (custom) datanode_1 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:173 - new 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 scm_1 | 2019-10-07 08:08:56 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol om_1 | 2019-10-07 08:09:07 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:173 - new 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) kdc_1 | WARNING: no policy specified for scm/311de0fd64d4@EXAMPLE.COM; defaulting to no policy datanode_3 | 2019-10-07 08:08:39 INFO RaftServerImpl:181 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: start as a follower, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.byte-limit = 2147483647 (custom) scm_1 | 2019-10-07 08:08:57 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.byte-limit = 2147483647 (custom) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) kdc_1 | Principal "scm/311de0fd64d4@EXAMPLE.COM" created. datanode_3 | 2019-10-07 08:08:39 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.element-limit = 1024 (custom) scm_1 | 2019-10-07 08:08:57 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:09:08 INFO OMVolumeCreateRequest:193 - created volume:33651-rpcwoport for user:testuser/scm@EXAMPLE.COM datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.queue.element-limit = 1024 (custom) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) datanode_3 | 2019-10-07 08:08:39 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start FollowerState datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) scm_1 | 2019-10-07 08:09:07 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:10 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.segment.size.max = 1048576 (custom) kdc_1 | Entry for principal scm/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.311de0fd64d4.keytab. s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) datanode_3 | 2019-10-07 08:08:39 INFO JmxRegister:44 - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-A621526D5DE4,id=8b31d836-555f-487d-9990-0cc47031ea16 datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.preallocated.size = 16384 (custom) scm_1 | 2019-10-07 08:09:07 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.preallocated.size = 16384 (custom) om_1 | 2019-10-07 08:09:10 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal scm/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.311de0fd64d4.keytab. s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) datanode_3 | 2019-10-07 08:08:43 INFO FollowerState:108 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE-FollowerState: change to CANDIDATE, lastRpcTime:5136ms, electionTimeout:5135ms datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.write.buffer.size = 33554432 (custom) scm_1 | 2019-10-07 08:09:09 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:13 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.write.buffer.size = 33554432 (custom) kdc_1 | Generiting keytab s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) datanode_3 | 2019-10-07 08:08:43 INFO RoleInfo:121 - 8b31d836-555f-487d-9990-0cc47031ea16: shutdown FollowerState datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.force.sync.num = 128 (default) scm_1 | 2019-10-07 08:09:09 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.force.sync.num = 128 (default) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync = true (default) scm_1 | 2019-10-07 08:09:09 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:09:16 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync = true (default) kdc_1 | WARNING: no policy specified for om/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) datanode_3 | 2019-10-07 08:08:43 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start LeaderElection datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout = 10s (default) scm_1 | 2019-10-07 08:09:09 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:09:16 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout = 10s (default) kdc_1 | Principal "om/ad18a6fa7980@EXAMPLE.COM" created. s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) datanode_3 | 2019-10-07 08:08:43 INFO LeaderElection:178 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE-LeaderElection1: begin an election at term 1 for -1: [8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858], old=null datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) scm_1 | 2019-10-07 08:09:27 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:19 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_3 | 2019-10-07 08:08:43 INFO RoleInfo:134 - 8b31d836-555f-487d-9990-0cc47031ea16: shutdown LeaderElection scm_1 | 2019-10-07 08:09:27 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:09:19 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.log.statemachine.data.caching.enabled = true (custom) kdc_1 | Entry for principal om/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.ad18a6fa7980.keytab. s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) datanode_1 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:128 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_3 | 2019-10-07 08:08:43 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: changes role from CANDIDATE to LEADER at term 1 for changeToLeader scm_1 | 2019-10-07 08:09:31 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:21 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO SegmentedRaftLogWorker:128 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 kdc_1 | Entry for principal om/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.ad18a6fa7980.keytab. s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerImpl:255 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: change Leader from null to 8b31d836-555f-487d-9990-0cc47031ea16 at term 1 for becomeLeader, leader elected after 5293ms scm_1 | 2019-10-07 08:09:31 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol om_1 | 2019-10-07 08:09:21 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.enabled = true (custom) kdc_1 | Generiting keytab s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerConfigKeys:43 - raft.server.staging.catchup.gap = 1000 (default) scm_1 | 2019-10-07 08:09:32 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:24 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.auto.trigger.threshold = 10000 (custom) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.retention.file.num = 5 (custom) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time = 25ms (default) scm_1 | 2019-10-07 08:09:32 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol om_1 | 2019-10-07 08:09:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.snapshot.retention.file.num = 5 (custom) kdc_1 | WARNING: no policy specified for om/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.retrycache.expirytime = 600000ms (custom) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerConfigKeys:43 - raft.server.write.element-limit = 4096 (default) scm_1 | 2019-10-07 08:09:33 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:09:27 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerConfigKeys:43 - raft.server.retrycache.expirytime = 600000ms (custom) kdc_1 | Principal "om/d8b9c0e14ce8@EXAMPLE.COM" created. s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:181 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: start as a follower, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null scm_1 | 2019-10-07 08:09:33 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:43 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout = 10s (default) om_1 | 2019-10-07 08:09:27 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RaftServerImpl:181 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: start as a follower, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_1 | 2019-10-07 08:08:39 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: changes role from null to FOLLOWER at term 0 for startAsFollower scm_1 | 2019-10-07 08:09:39 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout.denomination = 1s (default) om_1 | 2019-10-07 08:09:30 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: changes role from null to FOLLOWER at term 0 for startAsFollower s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) kdc_1 | Entry for principal om/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.d8b9c0e14ce8.keytab. datanode_1 | 2019-10-07 08:08:39 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start FollowerState scm_1 | 2019-10-07 08:09:39 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:43 INFO RaftServerConfigKeys:43 - raft.server.watch.element-limit = 65536 (default) om_1 | 2019-10-07 08:09:30 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:39 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start FollowerState s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) datanode_1 | 2019-10-07 08:08:39 INFO JmxRegister:44 - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-A621526D5DE4,id=8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765 scm_1 | 2019-10-07 08:09:57 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:43 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start LeaderState om_1 | 2019-10-07 08:09:35 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:39 INFO JmxRegister:44 - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-A621526D5DE4,id=27f27e32-1f1d-417d-8e7c-554581f7cfc6 kdc_1 | Entry for principal om/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.d8b9c0e14ce8.keytab. s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) datanode_1 | 2019-10-07 08:08:44 INFO FollowerState:108 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A-FollowerState: change to CANDIDATE, lastRpcTime:5083ms, electionTimeout:5082ms scm_1 | 2019-10-07 08:09:57 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:43 INFO SegmentedRaftLogWorker:385 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE-SegmentedRaftLogWorker: Starting segment from index:0 om_1 | 2019-10-07 08:09:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:44 INFO FollowerState:108 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4-FollowerState: change to CANDIDATE, lastRpcTime:5174ms, electionTimeout:5173ms kdc_1 | Generiting keytab s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) datanode_1 | 2019-10-07 08:08:44 INFO RoleInfo:121 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: shutdown FollowerState scm_1 | 2019-10-07 08:10:03 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:43 INFO RaftServerImpl:356 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE: set configuration 0: [8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858], old=null at 0 om_1 | 2019-10-07 08:09:40 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:44 INFO RoleInfo:121 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: shutdown FollowerState kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) datanode_1 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate scm_1 | 2019-10-07 08:10:03 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:43 INFO SegmentedRaftLogWorker:574 - 8b31d836-555f-487d-9990-0cc47031ea16@group-8345266181FE-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/b8de9a03-0aa9-4b04-82bb-8345266181fe/current/log_inprogress_0 om_1 | 2019-10-07 08:09:40 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate kdc_1 | WARNING: no policy specified for om/c8a6426e556a@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) datanode_1 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start LeaderElection scm_1 | 2019-10-07 08:10:09 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:44 INFO FollowerState:108 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-FollowerState: change to CANDIDATE, lastRpcTime:5023ms, electionTimeout:5023ms om_1 | 2019-10-07 08:09:43 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start LeaderElection kdc_1 | Principal "om/c8a6426e556a@EXAMPLE.COM" created. s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) datanode_1 | 2019-10-07 08:08:44 INFO LeaderElection:178 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A-LeaderElection1: begin an election at term 1 for -1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858], old=null scm_1 | 2019-10-07 08:10:09 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:44 INFO RoleInfo:121 - 8b31d836-555f-487d-9990-0cc47031ea16: shutdown FollowerState om_1 | 2019-10-07 08:09:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) datanode_2 | 2019-10-07 08:08:44 INFO LeaderElection:178 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4-LeaderElection1: begin an election at term 1 for -1: [27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null datanode_1 | 2019-10-07 08:08:44 INFO RoleInfo:134 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: shutdown LeaderElection scm_1 | 2019-10-07 08:10:27 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate om_1 | 2019-10-07 08:09:46 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Entry for principal om/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.c8a6426e556a.keytab. s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) datanode_2 | 2019-10-07 08:08:44 INFO RoleInfo:134 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: shutdown LeaderElection datanode_1 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: changes role from CANDIDATE to LEADER at term 1 for changeToLeader scm_1 | 2019-10-07 08:10:27 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start LeaderElection om_1 | 2019-10-07 08:09:46 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal om/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.c8a6426e556a.keytab. s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) datanode_2 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: changes role from CANDIDATE to LEADER at term 1 for changeToLeader datanode_1 | 2019-10-07 08:08:44 INFO RaftServerImpl:255 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: change Leader from null to 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765 at term 1 for becomeLeader, leader elected after 5269ms scm_1 | 2019-10-07 08:10:33 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:44 INFO LeaderElection:178 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-LeaderElection2: begin an election at term 1 for -1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null om_1 | 2019-10-07 08:09:48 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Generiting keytab s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) datanode_2 | 2019-10-07 08:08:44 INFO RaftServerImpl:255 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: change Leader from null to 27f27e32-1f1d-417d-8e7c-554581f7cfc6 at term 1 for becomeLeader, leader elected after 5302ms datanode_1 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.staging.catchup.gap = 1000 (default) scm_1 | 2019-10-07 08:10:33 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_3 | 2019-10-07 08:08:45 INFO LeaderElection:56 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-LeaderElection2: Election REJECTED; received 2 response(s) [8b31d836-555f-487d-9990-0cc47031ea16<-8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765#0:FAIL-t1, 8b31d836-555f-487d-9990-0cc47031ea16<-27f27e32-1f1d-417d-8e7c-554581f7cfc6#0:FAIL-t1] and 0 exception(s); 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4:t1, leader=null, voted=8b31d836-555f-487d-9990-0cc47031ea16, raftlog=8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) datanode_2 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.staging.catchup.gap = 1000 (default) datanode_1 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time = 25ms (default) scm_1 | 2019-10-07 08:10:37 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:08:45 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: changes role from CANDIDATE to FOLLOWER at term 1 for DISCOVERED_A_NEW_TERM kdc_1 | WARNING: no policy specified for om/s3g@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | 2019-10-07 08:09:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time = 25ms (default) scm_1 | 2019-10-07 08:10:37 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol datanode_1 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.write.element-limit = 4096 (default) datanode_3 | 2019-10-07 08:08:45 INFO RoleInfo:134 - 8b31d836-555f-487d-9990-0cc47031ea16: shutdown LeaderElection kdc_1 | Principal "om/s3g@EXAMPLE.COM" created. s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) datanode_2 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.write.element-limit = 4096 (default) om_1 | 2019-10-07 08:09:51 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:10:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 1 blocks datanode_1 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout = 10s (default) datanode_3 | 2019-10-07 08:08:45 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start FollowerState kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_2 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout = 10s (default) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | 2019-10-07 08:09:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:10:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 4 locID: 102920078705819773 bcsId: 0 datanode_1 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout.denomination = 1s (default) datanode_3 | 2019-10-07 08:08:50 INFO RaftServerImpl:170 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: changes role from FOLLOWER to FOLLOWER at term 2 for recognizeCandidate:27f27e32-1f1d-417d-8e7c-554581f7cfc6 kdc_1 | Entry for principal om/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.s3g.keytab. datanode_2 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout.denomination = 1s (default) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | 2019-10-07 08:09:54 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:10:39 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.watch.element-limit = 65536 (default) datanode_3 | 2019-10-07 08:08:50 INFO RoleInfo:121 - 8b31d836-555f-487d-9990-0cc47031ea16: shutdown FollowerState kdc_1 | Entry for principal om/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.s3g.keytab. datanode_2 | 2019-10-07 08:08:44 INFO RaftServerConfigKeys:43 - raft.server.watch.element-limit = 65536 (default) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | 2019-10-07 08:09:54 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:10:39 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_1 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start LeaderState datanode_3 | 2019-10-07 08:08:50 INFO RoleInfo:143 - 8b31d836-555f-487d-9990-0cc47031ea16: start FollowerState kdc_1 | Generiting keytab s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | 2019-10-07 08:09:57 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:44 INFO SegmentedRaftLogWorker:385 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A-SegmentedRaftLogWorker: Starting segment from index:0 scm_1 | 2019-10-07 08:10:48 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start LeaderState datanode_3 | 2019-10-07 08:08:50 INFO FollowerState:117 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | 2019-10-07 08:09:57 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_1 | 2019-10-07 08:08:44 INFO RaftServerImpl:356 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A: set configuration 0: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858], old=null at 0 scm_1 | 2019-10-07 08:10:48 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol datanode_2 | 2019-10-07 08:08:44 INFO SegmentedRaftLogWorker:385 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4-SegmentedRaftLogWorker: Starting segment from index:0 datanode_3 | 2019-10-07 08:08:50 INFO RaftServerImpl:255 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: change Leader from null to 27f27e32-1f1d-417d-8e7c-554581f7cfc6 at term 2 for appendEntries, leader elected after 10525ms kdc_1 | WARNING: no policy specified for scm/scm@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | 2019-10-07 08:10:00 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:44 INFO FollowerState:108 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-FollowerState: change to CANDIDATE, lastRpcTime:5037ms, electionTimeout:5037ms scm_1 | 2019-10-07 08:10:50 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:44 INFO RaftServerImpl:356 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4: set configuration 0: [27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null at 0 kdc_1 | Principal "scm/scm@EXAMPLE.COM" created. s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | 2019-10-07 08:10:00 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_1 | 2019-10-07 08:08:44 INFO RoleInfo:121 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: shutdown FollowerState datanode_3 | 2019-10-07 08:08:50 INFO RaftServerImpl:356 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4: set configuration 0: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null at 0 datanode_2 | 2019-10-07 08:08:44 INFO SegmentedRaftLogWorker:574 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-7BA479E1AFB4-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/65453df7-dd08-4275-9d28-7ba479e1afb4/current/log_inprogress_0 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) scm_1 | 2019-10-07 08:10:50 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:10:01 INFO OMVolumeCreateRequest:193 - created volume:33651-rpcwoport2 for user:testuser/scm@EXAMPLE.COM datanode_1 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_3 | 2019-10-07 08:08:50 INFO SegmentedRaftLogWorker:385 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-SegmentedRaftLogWorker: Starting segment from index:0 datanode_2 | 2019-10-07 08:08:44 INFO FollowerState:108 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-FollowerState: change to CANDIDATE, lastRpcTime:5033ms, electionTimeout:5033ms kdc_1 | Entry for principal scm/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.scm.keytab. s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) scm_1 | 2019-10-07 08:10:57 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:10:03 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start LeaderElection datanode_3 | 2019-10-07 08:08:50 INFO SegmentedRaftLogWorker:574 - 8b31d836-555f-487d-9990-0cc47031ea16@group-A621526D5DE4-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4/current/log_inprogress_0 datanode_2 | 2019-10-07 08:08:44 INFO RoleInfo:121 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: shutdown FollowerState kdc_1 | Entry for principal scm/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.scm.keytab. s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) scm_1 | 2019-10-07 08:10:57 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_1 | 2019-10-07 08:08:44 INFO LeaderElection:178 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-LeaderElection2: begin an election at term 1 for -1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null om_1 | 2019-10-07 08:10:03 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_3 | 2019-10-07 08:11:37 INFO DNCertificateClient:278 - Getting certificate with certSerialId:1099937157938208. datanode_2 | 2019-10-07 08:08:44 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate kdc_1 | Generiting keytab s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) scm_1 | 2019-10-07 08:11:09 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:44 INFO SegmentedRaftLogWorker:574 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-D6EEC858101A-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/8431418d-9c49-4a90-bf6b-d6eec858101a/current/log_inprogress_0 om_1 | 2019-10-07 08:10:06 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_3 | 2019-10-07 08:11:38 WARN GrpcClientProtocolService:134 - 0-OrderedRequestStreamObserver0: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close datanode_2 | 2019-10-07 08:08:44 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start LeaderElection kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) scm_1 | 2019-10-07 08:11:09 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:10:06 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:44 INFO LeaderElection:178 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-LeaderElection2: begin an election at term 1 for -1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null kdc_1 | WARNING: no policy specified for HTTP/om@EXAMPLE.COM; defaulting to no policy datanode_3 | 2019-10-07 08:18:07 WARN GrpcClientProtocolService:134 - 3-OrderedRequestStreamObserver3: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) datanode_1 | 2019-10-07 08:08:45 INFO LeaderElection:56 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-LeaderElection2: Election REJECTED; received 2 response(s) [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765<-8b31d836-555f-487d-9990-0cc47031ea16#0:FAIL-t1, 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765<-27f27e32-1f1d-417d-8e7c-554581f7cfc6#0:FAIL-t1] and 0 exception(s); 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4:t1, leader=null, voted=8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765, raftlog=8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null scm_1 | 2019-10-07 08:11:20 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:11:20 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Principal "HTTP/om@EXAMPLE.COM" created. datanode_2 | 2019-10-07 08:08:45 INFO LeaderElection:56 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-LeaderElection2: Election REJECTED; received 2 response(s) [27f27e32-1f1d-417d-8e7c-554581f7cfc6<-8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765#0:FAIL-t1, 27f27e32-1f1d-417d-8e7c-554581f7cfc6<-8b31d836-555f-487d-9990-0cc47031ea16#0:FAIL-t1] and 0 exception(s); 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4:t1, leader=null, voted=27f27e32-1f1d-417d-8e7c-554581f7cfc6, raftlog=27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) datanode_1 | 2019-10-07 08:08:45 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: changes role from CANDIDATE to FOLLOWER at term 1 for DISCOVERED_A_NEW_TERM kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. datanode_2 | 2019-10-07 08:08:45 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: changes role from CANDIDATE to FOLLOWER at term 1 for DISCOVERED_A_NEW_TERM scm_1 | 2019-10-07 08:11:27 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | 2019-10-07 08:10:09 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:45 INFO RoleInfo:134 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: shutdown LeaderElection kdc_1 | Entry for principal HTTP/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.om.keytab. scm_1 | 2019-10-07 08:11:27 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | 2019-10-07 08:10:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_1 | 2019-10-07 08:08:45 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start FollowerState kdc_1 | Entry for principal HTTP/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.om.keytab. kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:11:36 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:10:11 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:50 INFO RaftServerImpl:170 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: changes role from FOLLOWER to FOLLOWER at term 2 for recognizeCandidate:27f27e32-1f1d-417d-8e7c-554581f7cfc6 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. kdc_1 | WARNING: no policy specified for HTTP/311de0fd64d4@EXAMPLE.COM; defaulting to no policy scm_1 | 2019-10-07 08:11:36 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 2019-10-07 08:10:11 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_1 | 2019-10-07 08:08:50 INFO RoleInfo:121 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: shutdown FollowerState kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:11:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | 2019-10-07 08:10:14 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:50 INFO RoleInfo:143 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765: start FollowerState datanode_2 | 2019-10-07 08:08:45 INFO RoleInfo:134 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: shutdown LeaderElection scm_1 | 2019-10-07 08:11:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.protocol.SCMSecurityProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 2019-10-07 08:10:14 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_1 | 2019-10-07 08:08:50 INFO FollowerState:117 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-07 08:08:45 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start FollowerState datanode_2 | 2019-10-07 08:08:50 INFO FollowerState:108 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-FollowerState: change to CANDIDATE, lastRpcTime:5027ms, electionTimeout:5026ms scm_1 | 2019-10-07 08:11:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) om_1 | 2019-10-07 08:10:17 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:08:50 INFO RaftServerImpl:255 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: change Leader from null to 27f27e32-1f1d-417d-8e7c-554581f7cfc6 at term 2 for appendEntries, leader elected after 10528ms kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:50 INFO RoleInfo:121 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: shutdown FollowerState scm_1 | 2019-10-07 08:11:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) om_1 | 2019-10-07 08:10:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_1 | 2019-10-07 08:08:50 INFO RaftServerImpl:356 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4: set configuration 0: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null at 0 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_2 | 2019-10-07 08:08:50 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: changes role from FOLLOWER to CANDIDATE at term 1 for changeToCandidate scm_1 | 2019-10-07 08:11:50 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) datanode_1 | 2019-10-07 08:08:50 INFO SegmentedRaftLogWorker:385 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-SegmentedRaftLogWorker: Starting segment from index:0 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:50 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start LeaderElection om_1 | 2019-10-07 08:10:20 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:11:50 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) datanode_1 | 2019-10-07 08:08:50 INFO SegmentedRaftLogWorker:574 - 8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765@group-A621526D5DE4-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4/current/log_inprogress_0 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:50 INFO LeaderElection:178 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-LeaderElection3: begin an election at term 2 for -1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null om_1 | 2019-10-07 08:10:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) scm_1 | 2019-10-07 08:11:57 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:09:32 INFO DNCertificateClient:278 - Getting certificate with certSerialId:1099937157938208. kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:50 INFO LeaderElection:56 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-LeaderElection3: Election PASSED; received 1 response(s) [27f27e32-1f1d-417d-8e7c-554581f7cfc6<-8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765#0:OK-t2] and 0 exception(s); 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4:t2, leader=null, voted=27f27e32-1f1d-417d-8e7c-554581f7cfc6, raftlog=27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null om_1 | 2019-10-07 08:10:23 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) scm_1 | 2019-10-07 08:11:57 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_1 | 2019-10-07 08:09:33 WARN GrpcClientProtocolService:134 - 0-OrderedRequestStreamObserver0: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 datanode_2 | 2019-10-07 08:08:50 INFO RoleInfo:134 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: shutdown LeaderElection om_1 | 2019-10-07 08:10:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) scm_1 | 2019-10-07 08:12:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) datanode_1 | 2019-10-07 08:10:50 WARN GrpcClientProtocolService:134 - 1-OrderedRequestStreamObserver1: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:50 INFO RaftServerImpl:170 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: changes role from CANDIDATE to LEADER at term 2 for changeToLeader om_1 | 2019-10-07 08:10:26 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) scm_1 | 2019-10-07 08:12:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_1 | 2019-10-07 08:12:30 WARN GrpcClientProtocolService:134 - 2-OrderedRequestStreamObserver2: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 datanode_2 | 2019-10-07 08:08:50 INFO RaftServerImpl:255 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: change Leader from null to 27f27e32-1f1d-417d-8e7c-554581f7cfc6 at term 2 for becomeLeader, leader elected after 10437ms om_1 | 2019-10-07 08:10:26 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) scm_1 | 2019-10-07 08:12:20 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.staging.catchup.gap = 1000 (default) om_1 | 2019-10-07 08:10:29 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) scm_1 | 2019-10-07 08:12:20 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:10:29 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time = 25ms (default) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:12:27 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:10:29 ERROR OMBucketAddAclRequest:112 - Add acl [user:superuser1:rwxy[ACCESS]] to path /33651-rpcwoport2/bb1 failed, because acl already exist datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.write.element-limit = 4096 (default) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:12:27 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:10:31 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout = 10s (default) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:12:28 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout.denomination = 1s (default) om_1 | 2019-10-07 08:10:31 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:12:28 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.watch.element-limit = 65536 (default) om_1 | 2019-10-07 08:10:34 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:12:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 1 blocks s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:10:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:12:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 6 locID: 102920086890938495 bcsId: 0 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) om_1 | 2019-10-07 08:10:37 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:12:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) om_1 | 2019-10-07 08:10:37 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:12:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 2019-10-07 08:10:40 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.element-limit = 1 (custom) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:12:57 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 2019-10-07 08:10:40 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:12:57 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) datanode_2 | 2019-10-07 08:08:50 INFO GrpcConfigKeys$Server:43 - raft.grpc.server.leader.outstanding.appends.max = 128 (default) om_1 | 2019-10-07 08:10:43 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:13:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.rpc.request.timeout = 3000ms (default) om_1 | 2019-10-07 08:10:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:13:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 2019-10-07 08:10:48 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:13:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | 2019-10-07 08:10:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:13:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | 2019-10-07 08:10:52 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:13:27 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.element-limit = 1 (custom) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | 2019-10-07 08:10:52 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:13:27 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | 2019-10-07 08:10:55 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:13:30 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:50 INFO GrpcConfigKeys$Server:43 - raft.grpc.server.leader.outstanding.appends.max = 128 (default) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | 2019-10-07 08:10:55 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:13:30 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.rpc.request.timeout = 3000ms (default) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 2019-10-07 08:10:58 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:13:37 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerConfigKeys:43 - raft.server.log.appender.install.snapshot.enabled = false (custom) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | 2019-10-07 08:10:58 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:13:37 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) datanode_2 | 2019-10-07 08:08:50 INFO RoleInfo:143 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6: start LeaderState kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:13:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 1 blocks s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) datanode_2 | 2019-10-07 08:08:50 INFO SegmentedRaftLogWorker:385 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-SegmentedRaftLogWorker: Starting segment from index:0 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:11:01 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:13:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 7 locID: 102920090336034944 bcsId: 0 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) datanode_2 | 2019-10-07 08:08:50 INFO RaftServerImpl:356 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4: set configuration 0: [8c3fdd1b-0b96-4e9e-a5e1-7823aac6e765:172.18.0.8:9858, 8b31d836-555f-487d-9990-0cc47031ea16:172.18.0.10:9858, 27f27e32-1f1d-417d-8e7c-554581f7cfc6:172.18.0.5:9858], old=null at 0 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:11:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:13:37 INFO ReplicationManager:162 - Starting Replication Monitor Thread. s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) datanode_2 | 2019-10-07 08:08:50 INFO SegmentedRaftLogWorker:574 - 27f27e32-1f1d-417d-8e7c-554581f7cfc6@group-A621526D5DE4-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/c0883aad-232c-4e37-8411-a621526d5de4/current/log_inprogress_0 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:11:04 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:13:37 INFO ReplicationManager:225 - Replication Monitor Thread took 12 milliseconds for processing 7 containers. s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) datanode_2 | 2019-10-07 08:08:56 INFO DNCertificateClient:278 - Getting certificate with certSerialId:1099937157938208. kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:11:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:13:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) datanode_2 | 2019-10-07 08:14:07 WARN GrpcClientProtocolService:134 - 0-OrderedRequestStreamObserver0: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:11:07 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) scm_1 | 2019-10-07 08:13:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:11:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) scm_1 | 2019-10-07 08:13:57 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:11:09 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) scm_1 | 2019-10-07 08:13:57 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:11:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:14:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:11:12 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:14:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:11:12 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:14:05 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:11:13 INFO OMVolumeCreateRequest:193 - created volume:33651-rpcwport for user:testuser/scm@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:14:05 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol om_1 | 2019-10-07 08:11:15 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:14:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:15 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) scm_1 | 2019-10-07 08:14:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:18 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) scm_1 | 2019-10-07 08:14:30 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:18 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:14:30 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:20 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:14:36 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:14:36 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:23 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:14:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:14:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:26 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:15:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:26 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) scm_1 | 2019-10-07 08:15:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:11:29 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) scm_1 | 2019-10-07 08:15:06 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:29 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) scm_1 | 2019-10-07 08:15:06 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:32 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 2019-10-07 08:11:32 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:15:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:11:35 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:15:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:11:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:15:30 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 2019-10-07 08:11:40 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:15:30 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:40 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) scm_1 | 2019-10-07 08:15:36 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:45 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:15:36 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:15:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:48 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:15:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:11:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:15:39 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:11:51 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:11:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) scm_1 | 2019-10-07 08:15:39 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | 2019-10-07 08:11:53 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:16:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | Oct 07, 2019 8:15:47 AM org.glassfish.jersey.internal.Errors logErrors om_1 | 2019-10-07 08:11:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected: om_1 | 2019-10-07 08:11:56 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:16:02 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | MultiException stack 1 of 1 om_1 | 2019-10-07 08:11:56 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:02 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:11:59 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:16:06 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](info): closing down fd 18 s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | 2019-10-07 08:11:59 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:06 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | 2019-10-07 08:12:02 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:16:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) om_1 | 2019-10-07 08:12:02 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:12:05 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) scm_1 | 2019-10-07 08:16:27 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:12:05 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:27 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | 2019-10-07 08:12:05 INFO OMVolumeCreateRequest:193 - created volume:33651-rpcwoscheme for user:testuser/scm@EXAMPLE.COM scm_1 | 2019-10-07 08:16:28 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | 2019-10-07 08:12:07 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:16:28 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Principal "HTTP/311de0fd64d4@EXAMPLE.COM" created. s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | 2019-10-07 08:12:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:30 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:12:10 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) scm_1 | 2019-10-07 08:16:30 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Entry for principal HTTP/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.311de0fd64d4.keytab. om_1 | 2019-10-07 08:12:10 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:16:36 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) kdc_1 | Entry for principal HTTP/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.311de0fd64d4.keytab. om_1 | 2019-10-07 08:12:13 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:16:36 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:16:54 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) scm_1 | 2019-10-07 08:16:54 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | 2019-10-07 08:12:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) scm_1 | 2019-10-07 08:16:58 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:12:16 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | WARNING: no policy specified for scm/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy om_1 | 2019-10-07 08:12:16 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) scm_1 | 2019-10-07 08:16:58 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Principal "scm/ad18a6fa7980@EXAMPLE.COM" created. om_1 | 2019-10-07 08:12:19 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) scm_1 | 2019-10-07 08:17:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:12:19 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) scm_1 | 2019-10-07 08:17:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:22 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:17:06 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:12:22 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:17:06 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:25 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:17:21 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:17:21 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol om_1 | 2019-10-07 08:12:25 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:17:28 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:12:27 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:17:28 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:12:27 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) scm_1 | 2019-10-07 08:17:30 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:32 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) scm_1 | 2019-10-07 08:17:30 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:12:32 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:17:36 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:37 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:17:36 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:17:37 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:37 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:17:37 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:12:40 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:17:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 3 blocks om_1 | 2019-10-07 08:12:40 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:17:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 9 locID: 102920105975742594 bcsId: 0 om_1 | 2019-10-07 08:12:42 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:17:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 10 locID: 102920106480500867 bcsId: 0 om_1 | 2019-10-07 08:12:42 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) scm_1 | 2019-10-07 08:17:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 4 locID: 102920107754127492 bcsId: 0 om_1 | 2019-10-07 08:12:45 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) scm_1 | 2019-10-07 08:17:55 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:12:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) scm_1 | 2019-10-07 08:17:55 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:48 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) scm_1 | 2019-10-07 08:18:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:12:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) scm_1 | 2019-10-07 08:18:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:51 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:18:06 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | 2019-10-07 08:12:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:18:06 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 2019-10-07 08:12:54 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:12:54 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:18:08 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | 2019-10-07 08:13:01 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:18:08 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:13:02 INFO OMVolumeCreateRequest:193 - created volume:fstest97 for user:testuser/scm@EXAMPLE.COM scm_1 | 2019-10-07 08:18:30 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:13:04 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:18:30 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:13:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:18:36 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:13:05 INFO OMVolumeCreateRequest:193 - created volume:fstest297 for user:testuser/scm@EXAMPLE.COM scm_1 | 2019-10-07 08:18:36 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) kdc_1 | Oct 07 08:08:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435702, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:18:37 INFO ReplicationManager:225 - Replication Monitor Thread took 1 milliseconds for processing 12 containers. s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) scm_1 | 2019-10-07 08:18:38 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | 2019-10-07 08:13:07 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:18:38 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:18:53 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:13:10 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:18:53 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol om_1 | 2019-10-07 08:13:10 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:19:00 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:13:13 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database scm_1 | 2019-10-07 08:19:00 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM scm_1 | 2019-10-07 08:19:02 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:13:16 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:19:02 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol om_1 | 2019-10-07 08:13:16 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) kdc_1 | Entry for principal scm/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.ad18a6fa7980.keytab. scm_1 | 2019-10-07 08:19:04 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:13:18 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) kdc_1 | Entry for principal scm/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.ad18a6fa7980.keytab. scm_1 | 2019-10-07 08:19:04 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:13:18 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:19:04 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:19 INFO OMVolumeCreateRequest:193 - created volume:fstest397 for user:testuser/scm@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:19:04 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:21 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) kdc_1 | WARNING: no policy specified for scm/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy scm_1 | 2019-10-07 08:19:35 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | 2019-10-07 08:13:21 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Principal "scm/d8b9c0e14ce8@EXAMPLE.COM" created. scm_1 | 2019-10-07 08:19:35 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | 2019-10-07 08:13:24 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:19:35 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | 2019-10-07 08:13:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal scm/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.d8b9c0e14ce8.keytab. scm_1 | 2019-10-07 08:19:35 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) kdc_1 | Entry for principal scm/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.d8b9c0e14ce8.keytab. scm_1 | 2019-10-07 08:19:35 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:27 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) kdc_1 | Generiting keytab om_1 | 2019-10-07 08:13:27 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) scm_1 | 2019-10-07 08:19:35 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:13:30 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) kdc_1 | WARNING: no policy specified for scm/c8a6426e556a@EXAMPLE.COM; defaulting to no policy om_1 | 2019-10-07 08:13:30 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:19:37 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Principal "scm/c8a6426e556a@EXAMPLE.COM" created. s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 2019-10-07 08:13:33 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:19:37 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 2019-10-07 08:13:33 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal scm/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.c8a6426e556a.keytab. scm_1 | 2019-10-07 08:20:07 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 2019-10-07 08:13:36 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Entry for principal scm/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.c8a6426e556a.keytab. scm_1 | 2019-10-07 08:20:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:13:36 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:20:07 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 2019-10-07 08:13:39 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:20:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 2019-10-07 08:13:39 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | WARNING: no policy specified for scm/s3g@EXAMPLE.COM; defaulting to no policy scm_1 | 2019-10-07 08:20:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | 2019-10-07 08:13:42 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Principal "scm/s3g@EXAMPLE.COM" created. scm_1 | 2019-10-07 08:20:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:13:42 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:20:32 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:13:45 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Entry for principal scm/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.s3g.keytab. scm_1 | 2019-10-07 08:20:32 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:13:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal scm/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.s3g.keytab. scm_1 | 2019-10-07 08:20:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 3 blocks s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | 2019-10-07 08:13:47 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:20:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 15 locID: 102920118418669709 bcsId: 0 s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | 2019-10-07 08:13:47 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:20:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 14 locID: 102920122045563029 bcsId: 0 s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | 2019-10-07 08:13:48 ERROR OMBucketAddAclRequest:112 - Add acl [user:superuser1:rwxy[ACCESS]] to path /fstest397/bk1 failed, because acl already exist kdc_1 | WARNING: no policy specified for HTTP/scm@EXAMPLE.COM; defaulting to no policy scm_1 | 2019-10-07 08:20:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 15 locID: 102920122105987222 bcsId: 0 s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | 2019-10-07 08:13:50 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Principal "HTTP/scm@EXAMPLE.COM" created. scm_1 | 2019-10-07 08:20:37 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | 2019-10-07 08:13:50 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:20:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | 2019-10-07 08:13:53 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Entry for principal HTTP/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.scm.keytab. scm_1 | 2019-10-07 08:20:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | 2019-10-07 08:13:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:20:37 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Entry for principal HTTP/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.scm.keytab. s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:13:56 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:20:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Generiting keytab s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:13:56 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:20:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:13:59 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) scm_1 | 2019-10-07 08:21:07 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:13:59 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | WARNING: no policy specified for testuser/om@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) scm_1 | 2019-10-07 08:21:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:14:01 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Principal "testuser/om@EXAMPLE.COM" created. s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:14:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) kdc_1 | Entry for principal testuser/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.om.keytab. om_1 | 2019-10-07 08:14:04 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) kdc_1 | Entry for principal testuser/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.om.keytab. scm_1 | 2019-10-07 08:21:07 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:14:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) scm_1 | 2019-10-07 08:21:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Generiting keytab s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) scm_1 | 2019-10-07 08:21:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:14:09 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) scm_1 | 2019-10-07 08:21:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | WARNING: no policy specified for testuser/311de0fd64d4@EXAMPLE.COM; defaulting to no policy om_1 | 2019-10-07 08:14:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:21:22 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) kdc_1 | Principal "testuser/311de0fd64d4@EXAMPLE.COM" created. om_1 | 2019-10-07 08:14:12 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:21:22 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:14:12 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:21:37 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) kdc_1 | Entry for principal testuser/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.311de0fd64d4.keytab. om_1 | 2019-10-07 08:14:15 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:21:37 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) kdc_1 | Entry for principal testuser/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.311de0fd64d4.keytab. om_1 | 2019-10-07 08:14:15 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) scm_1 | 2019-10-07 08:21:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 1 blocks kdc_1 | Generiting keytab om_1 | 2019-10-07 08:14:17 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol scm_1 | 2019-10-07 08:21:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 14 locID: 102920122729758872 bcsId: 0,conID: 15 locID: 102920122730872985 bcsId: 0,conID: 13 locID: 102920122729627799 bcsId: 0 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | 2019-10-07 08:14:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:21:37 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | WARNING: no policy specified for HTTP/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | 2019-10-07 08:14:20 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:21:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Principal "HTTP/ad18a6fa7980@EXAMPLE.COM" created. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | 2019-10-07 08:14:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:21:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:14:23 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:21:37 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | 2019-10-07 08:14:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal HTTP/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.ad18a6fa7980.keytab. scm_1 | 2019-10-07 08:21:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) kdc_1 | Entry for principal HTTP/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.ad18a6fa7980.keytab. om_1 | 2019-10-07 08:14:26 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:21:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) kdc_1 | Generiting keytab om_1 | 2019-10-07 08:14:26 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:21:51 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:14:29 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:21:51 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol kdc_1 | WARNING: no policy specified for HTTP/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:14:29 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:22:07 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Principal "HTTP/d8b9c0e14ce8@EXAMPLE.COM" created. s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | 2019-10-07 08:14:32 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:22:07 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | ... 92 more scm_1 | 2019-10-07 08:22:07 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:14:32 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Entry for principal HTTP/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.d8b9c0e14ce8.keytab. s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 scm_1 | 2019-10-07 08:22:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:14:32 WARN OzoneManager:1621 - User testuser2/scm@EXAMPLE.COM doesn't have READ permission to access volume s3g_1 | 20191007T081546Z kdc_1 | Entry for principal HTTP/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.d8b9c0e14ce8.keytab. scm_1 | 2019-10-07 08:22:07 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:14:34 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Generiting keytab scm_1 | 2019-10-07 08:22:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. scm_1 | 2019-10-07 08:22:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | 2019-10-07 08:14:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:22:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | 2019-10-07 08:14:35 WARN OzoneManager:1621 - User testuser2/scm@EXAMPLE.COM doesn't have READ permission to access volume s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) kdc_1 | WARNING: no policy specified for HTTP/c8a6426e556a@EXAMPLE.COM; defaulting to no policy om_1 | 2019-10-07 08:14:37 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) scm_1 | 2019-10-07 08:22:31 INFO Server:1849 - Auth successful for om/om@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) kdc_1 | Principal "HTTP/c8a6426e556a@EXAMPLE.COM" created. om_1 | 2019-10-07 08:14:37 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol scm_1 | 2019-10-07 08:22:31 INFO ServiceAuthorizationManager:138 - Authorization successful for om/om@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:14:38 WARN OzoneManager:1621 - User testuser2/scm@EXAMPLE.COM doesn't have WRITE_ACL permission to access volume scm_1 | 2019-10-07 08:22:37 INFO SCMBlockProtocolServer:212 - SCM is informed by OM to delete 5 blocks s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:14:38 ERROR OMVolumeAddAclRequest:106 - Add acl user:testuser2/scm@EXAMPLE.COM:xy[ACCESS] to volume fstest397 failed! scm_1 | 2019-10-07 08:22:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 15 locID: 102920127247351970 bcsId: 0 s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User testuser2/scm@EXAMPLE.COM doesn't have WRITE_ACL permission to access volume scm_1 | 2019-10-07 08:22:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 13 locID: 102920128241139875 bcsId: 0 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1623) scm_1 | 2019-10-07 08:22:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 14 locID: 102920128837517476 bcsId: 0 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:135) scm_1 | 2019-10-07 08:22:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 15 locID: 102920129846771877 bcsId: 0 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ozone.om.request.volume.acl.OMVolumeAclRequest.validateAndUpdateCache(OMVolumeAclRequest.java:79) scm_1 | 2019-10-07 08:22:37 INFO BlockManagerImpl:269 - Deleting blocks conID: 13 locID: 102920129901625510 bcsId: 0 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) scm_1 | 2019-10-07 08:22:37 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) scm_1 | 2019-10-07 08:22:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) scm_1 | 2019-10-07 08:22:37 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) scm_1 | 2019-10-07 08:22:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) scm_1 | 2019-10-07 08:22:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) scm_1 | 2019-10-07 08:22:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) scm_1 | 2019-10-07 08:23:07 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) scm_1 | 2019-10-07 08:23:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:23:07 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:23:07 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:23:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | 2019-10-07 08:14:40 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:23:07 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | 2019-10-07 08:14:40 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:23:37 INFO ReplicationManager:225 - Replication Monitor Thread took 1 milliseconds for processing 15 containers. om_1 | 2019-10-07 08:14:43 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | ... 101 more kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:23:37 INFO Server:1849 - Auth successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:14:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 scm_1 | 2019-10-07 08:23:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/311de0fd64d4@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:14:44 WARN OzoneManager:1621 - User testuser2/scm@EXAMPLE.COM doesn't have LIST permission to access volume s3g_1 | kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:23:37 INFO Server:1849 - Auth successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:14:46 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 2019-10-07 08:15:47 WARN ServletHandler:625 - kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 scm_1 | 2019-10-07 08:23:37 INFO Server:1849 - Auth successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:14:46 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | javax.servlet.ServletException: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 scm_1 | 2019-10-07 08:23:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/ad18a6fa7980@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol om_1 | 2019-10-07 08:14:49 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 1. javax.enterprise.inject.CreationException scm_1 | 2019-10-07 08:23:37 INFO ServiceAuthorizationManager:138 - Authorization successful for dn/d8b9c0e14ce8@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:14:49 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:14:51 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:14:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:14:54 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 2019-10-07 08:14:54 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:14:55 WARN OzoneManager:1621 - User testuser2/scm@EXAMPLE.COM doesn't have READ permission to access bucket kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:14:57 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:14:57 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:15:00 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:00 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:15:03 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:15:03 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:15:06 INFO Server:1849 - Auth successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | 2019-10-07 08:15:06 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser2/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | 2019-10-07 08:15:39 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | 2019-10-07 08:15:39 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | 2019-10-07 08:15:43 INFO AWSV4AuthValidator:98 - fe9646975a46fab9f23504aeb1f9b362de4f81c4c01e169aaadfd687b9517a6f s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:15:43 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:TOKEN) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:15:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 2019-10-07 08:15:44 INFO AWSV4AuthValidator:98 - fe9646975a46fab9f23504aeb1f9b362de4f81c4c01e169aaadfd687b9517a6f s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:44 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:15:44 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:15:46 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | 1. javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:46 WARN Server:1833 - Auth failed for 172.18.0.9:43586:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081546Z s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:46 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:23 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) kdc_1 | Oct 07 08:08:23 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) kdc_1 | Entry for principal HTTP/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.c8a6426e556a.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) kdc_1 | Entry for principal HTTP/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.c8a6426e556a.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | ... 33 more kdc_1 | Generiting keytab s3g_1 | Caused by: javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) kdc_1 | WARNING: no policy specified for HTTP/s3g@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) kdc_1 | Principal "HTTP/s3g@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) om_1 | 2019-10-07 08:15:46 WARN Server:1833 - Auth failed for 172.18.0.9:43588:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:15:46 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) kdc_1 | Oct 07 08:08:23 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435703, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:15:46 WARN Server:1833 - Auth failed for 172.18.0.9:43590:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) om_1 | 2019-10-07 08:15:46 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | ... 60 more kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 20191007T081546Z s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Entry for principal HTTP/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.s3g.keytab. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) kdc_1 | Entry for principal HTTP/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.s3g.keytab. om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | WARNING: no policy specified for testuser/scm@EXAMPLE.COM; defaulting to no policy s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Principal "testuser/scm@EXAMPLE.COM" created. s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Entry for principal testuser/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.scm.keytab. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Entry for principal testuser/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.scm.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) kdc_1 | Generiting keytab om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | ... 92 more kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | WARNING: no policy specified for testuser2/om@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | 20191007T081546Z om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Principal "testuser2/om@EXAMPLE.COM" created. s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) kdc_1 | Entry for principal testuser2/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.om.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) kdc_1 | Entry for principal testuser2/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.om.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Generiting keytab s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) kdc_1 | WARNING: no policy specified for testuser2/311de0fd64d4@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) kdc_1 | Principal "testuser2/311de0fd64d4@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:15:46 WARN Server:1833 - Auth failed for 172.18.0.9:43592:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | 20191007T081546Z kdc_1 | Entry for principal testuser2/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.311de0fd64d4.keytab. s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal testuser2/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.311de0fd64d4.keytab. s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) kdc_1 | Generiting keytab om_1 | 2019-10-07 08:15:46 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 20191007T081546Z s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) kdc_1 | WARNING: no policy specified for testuser/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Principal "testuser/ad18a6fa7980@EXAMPLE.COM" created. s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Entry for principal testuser/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.ad18a6fa7980.keytab. s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Entry for principal testuser/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.ad18a6fa7980.keytab. om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) kdc_1 | WARNING: no policy specified for testuser/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Principal "testuser/d8b9c0e14ce8@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) kdc_1 | Entry for principal testuser/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.d8b9c0e14ce8.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) kdc_1 | Entry for principal testuser/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.d8b9c0e14ce8.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | ... 101 more om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | WARNING: no policy specified for testuser/c8a6426e556a@EXAMPLE.COM; defaulting to no policy s3g_1 | 2019-10-07 08:15:47 WARN HttpChannel:499 - //s3g:9878/bucket-test123 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Principal "testuser/c8a6426e556a@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Entry for principal testuser/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.c8a6426e556a.keytab. s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Entry for principal testuser/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.c8a6426e556a.keytab. s3g_1 | om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Generiting keytab s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) kdc_1 | WARNING: no policy specified for testuser/s3g@EXAMPLE.COM; defaulting to no policy om_1 | 2019-10-07 08:15:46 WARN Server:1833 - Auth failed for 172.18.0.9:43594:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) kdc_1 | Principal "testuser/s3g@EXAMPLE.COM" created. om_1 | 20191007T081546Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal testuser/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.s3g.keytab. s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) kdc_1 | Entry for principal testuser/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.s3g.keytab. s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Generiting keytab s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 20191007T081546Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) kdc_1 | WARNING: no policy specified for testuser2/scm@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) kdc_1 | Principal "testuser2/scm@EXAMPLE.COM" created. kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) kdc_1 | Entry for principal testuser2/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.scm.keytab. kdc_1 | Entry for principal testuser2/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.scm.keytab. s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | Caused by: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | 1. javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43596:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081546Z s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | ... 13 more kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | 1. javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43598:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081546Z s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | ... 33 more kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | Caused by: javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43600:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | 20191007T081546Z s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) kdc_1 | Oct 07 08:08:24 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) kdc_1 | Generiting keytab om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | WARNING: no policy specified for s3g/om@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) kdc_1 | Principal "s3g/om@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43602:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 20191007T081546Z s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | ... 60 more om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 20191007T081546Z s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | ... 92 more kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | 20191007T081546Z om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:24 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435704, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43604:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | 20191007T081546Z kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal s3g/om@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.om.keytab. s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) kdc_1 | Entry for principal s3g/om@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.om.keytab. om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) kdc_1 | Generiting keytab om_1 | 20191007T081546Z s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) kdc_1 | WARNING: no policy specified for s3g/311de0fd64d4@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) kdc_1 | Principal "s3g/311de0fd64d4@EXAMPLE.COM" created. om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Entry for principal s3g/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.311de0fd64d4.keytab. s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Entry for principal s3g/311de0fd64d4@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.311de0fd64d4.keytab. om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Generiting keytab s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | WARNING: no policy specified for testuser2/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Principal "testuser2/ad18a6fa7980@EXAMPLE.COM" created. s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Entry for principal testuser2/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.ad18a6fa7980.keytab. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Entry for principal testuser2/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.ad18a6fa7980.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | ... 101 more kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | 20191007T081547Z kdc_1 | WARNING: no policy specified for testuser2/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Principal "testuser2/d8b9c0e14ce8@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | 20191007T081547Z kdc_1 | Entry for principal testuser2/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.d8b9c0e14ce8.keytab. om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43606:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal testuser2/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.d8b9c0e14ce8.keytab. om_1 | 20191007T081546Z kdc_1 | Generiting keytab s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | WARNING: no policy specified for testuser2/c8a6426e556a@EXAMPLE.COM; defaulting to no policy s3g_1 | 20191007T081547Z om_1 | ca146df73ceee5b07fbb917db23a24e09f8a8769a7910966172815bcb0f6a32a, signature=c13bf5697551f9f2b43f52d4512c0454cb753a2301b00a27780aed33c17fec47, awsAccessKeyId=dlfknslnfslf) kdc_1 | Principal "testuser2/c8a6426e556a@EXAMPLE.COM" created. s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover immediately. kdc_1 | Entry for principal testuser2/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.c8a6426e556a.keytab. om_1 | 20191007T081547Z s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Entry for principal testuser2/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.c8a6426e556a.keytab. om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 20191007T081547Z kdc_1 | Generiting keytab om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | WARNING: no policy specified for testuser2/s3g@EXAMPLE.COM; defaulting to no policy s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Principal "testuser2/s3g@EXAMPLE.COM" created. s3g_1 | 20191007T081547Z om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Entry for principal testuser2/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.s3g.keytab. s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 2 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Entry for principal testuser2/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.s3g.keytab. s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Generiting keytab s3g_1 | 20191007T081547Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | WARNING: no policy specified for s3g/scm@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Principal "s3g/scm@EXAMPLE.COM" created. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Entry for principal s3g/scm@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.scm.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Entry for principal s3g/scm@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.scm.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | 20191007T081547Z kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 3 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | WARNING: no policy specified for s3g/ad18a6fa7980@EXAMPLE.COM; defaulting to no policy s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Principal "s3g/ad18a6fa7980@EXAMPLE.COM" created. s3g_1 | 20191007T081547Z om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Entry for principal s3g/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.ad18a6fa7980.keytab. s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Entry for principal s3g/ad18a6fa7980@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.ad18a6fa7980.keytab. s3g_1 | 20191007T081547Z om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43610:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Generiting keytab s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 20191007T081547Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 4 failover attempts. Trying to failover immediately. om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | WARNING: no policy specified for s3g/d8b9c0e14ce8@EXAMPLE.COM; defaulting to no policy s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) kdc_1 | Principal "s3g/d8b9c0e14ce8@EXAMPLE.COM" created. om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081547Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | 20191007T081547Z s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal s3g/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.d8b9c0e14ce8.keytab. om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Entry for principal s3g/d8b9c0e14ce8@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.d8b9c0e14ce8.keytab. s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Generiting keytab om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | 20191007T081547Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | WARNING: no policy specified for s3g/c8a6426e556a@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 5 failover attempts. Trying to failover immediately. kdc_1 | Principal "s3g/c8a6426e556a@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | 20191007T081547Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal s3g/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.c8a6426e556a.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Entry for principal s3g/c8a6426e556a@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.c8a6426e556a.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | 20191007T081547Z kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | WARNING: no policy specified for s3g/s3g@EXAMPLE.COM; defaulting to no policy s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 6 failover attempts. Trying to failover immediately. kdc_1 | Principal "s3g/s3g@EXAMPLE.COM" created. s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | 20191007T081547Z om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | 20191007T081547Z om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 7 failover attempts. Trying to failover immediately. kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/om@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43612:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081547Z om_1 | 20191007T081547Z s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007T081547Z om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 8 failover attempts. Trying to failover immediately. om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/311de0fd64d4@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 2019-10-07 08:15:47 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | 20191007T081547Z om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 9 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 2019-10-07 08:15:47 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | 20191007T081547Z om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 2019-10-07 08:15:47 ERROR OMFailoverProxyProvider:268 - Failed to connect to OM. Attempted 10 retries and 10 failovers kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | 2019-10-07 08:15:47 ERROR OzoneClientFactory:259 - Couldn't create RpcClient protocol exception: s3g_1 | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | 20191007T081547Z om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43614:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007T081547Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | 20191007T081547Z s3g_1 | Oct 07, 2019 8:15:47 AM org.glassfish.jersey.internal.Errors logErrors om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected: om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | MultiException stack 1 of 1 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | javax.enterprise.inject.CreationException om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/scm@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/ad18a6fa7980@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43616:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/d8b9c0e14ce8@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | 20191007T081547Z s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:25 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/c8a6426e556a@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) kdc_1 | Oct 07 08:08:25 kdc kadmind[15](info): closing down fd 18 om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:26 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:26 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) kdc_1 | Oct 07 08:08:26 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) kdc_1 | Oct 07 08:08:26 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) kdc_1 | Entry for principal s3g/s3g@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.s3g.keytab. om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) kdc_1 | Entry for principal s3g/s3g@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.s3g.keytab. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43618:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 20191007T081547Z s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 20191007T081547Z s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43620:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | 20191007T081547Z s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435705, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) kdc_1 | Oct 07 08:08:26 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) kdc_1 | Oct 07 08:08:26 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435706, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:26 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:26 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:26 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/s3g@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:26 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:27 kdc krb5kdc[9](info): AS_REQ (2 etypes {18 17}) 172.18.0.2: ISSUE: authtime 1570435707, etypes {rep=18 tkt=18 ses=18}, om/om@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:27 kdc krb5kdc[9](info): AS_REQ (2 etypes {18 17}) 172.18.0.5: ISSUE: authtime 1570435707, etypes {rep=18 tkt=18 ses=18}, dn/311de0fd64d4@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) kdc_1 | Oct 07 08:08:27 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435707, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) kdc_1 | Oct 07 08:08:28 kdc krb5kdc[9](info): AS_REQ (2 etypes {18 17}) 172.18.0.8: ISSUE: authtime 1570435708, etypes {rep=18 tkt=18 ses=18}, dn/ad18a6fa7980@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) kdc_1 | Oct 07 08:08:28 kdc krb5kdc[9](info): AS_REQ (2 etypes {18 17}) 172.18.0.10: ISSUE: authtime 1570435708, etypes {rep=18 tkt=18 ses=18}, dn/d8b9c0e14ce8@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) kdc_1 | Oct 07 08:08:28 kdc krb5kdc[9](info): AS_REQ (2 etypes {18 17}) 172.18.0.7: ISSUE: authtime 1570435708, etypes {rep=18 tkt=18 ses=18}, scm/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) kdc_1 | Oct 07 08:08:30 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435710, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) kdc_1 | Oct 07 08:08:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.5: ISSUE: authtime 1570435707, etypes {rep=18 tkt=18 ses=18}, dn/311de0fd64d4@EXAMPLE.COM for scm/scm@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:08:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.10: ISSUE: authtime 1570435708, etypes {rep=18 tkt=18 ses=18}, dn/d8b9c0e14ce8@EXAMPLE.COM for scm/scm@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.8: ISSUE: authtime 1570435708, etypes {rep=18 tkt=18 ses=18}, dn/ad18a6fa7980@EXAMPLE.COM for scm/scm@EXAMPLE.COM s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.2: ISSUE: authtime 1570435707, etypes {rep=18 tkt=18 ses=18}, om/om@EXAMPLE.COM for scm/scm@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:32 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435712, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) kdc_1 | Oct 07 08:08:32 kdc krb5kdc[9](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435712, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for HTTP/scm@EXAMPLE.COM kdc_1 | Oct 07 08:08:35 kdc krb5kdc[9](info): AS_REQ (2 etypes {18 17}) 172.18.0.2: ISSUE: authtime 1570435715, etypes {rep=18 tkt=18 ses=18}, om/om@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM kdc_1 | Oct 07 08:08:35 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435715, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) kdc_1 | Oct 07 08:08:35 kdc krb5kdc[9](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435715, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for HTTP/scm@EXAMPLE.COM kdc_1 | Oct 07 08:08:35 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.2: ISSUE: authtime 1570435715, etypes {rep=18 tkt=18 ses=18}, om/om@EXAMPLE.COM for scm/scm@EXAMPLE.COM kdc_1 | Oct 07 08:08:37 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435717, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM kdc_1 | Oct 07 08:08:38 kdc krb5kdc[9](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435717, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for HTTP/scm@EXAMPLE.COM kdc_1 | Generiting keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) kdc_1 | WARNING: no policy specified for dn/recon@EXAMPLE.COM; defaulting to no policy om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Principal "dn/recon@EXAMPLE.COM" created. s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) kdc_1 | Entry for principal dn/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.recon.keytab. om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) kdc_1 | Entry for principal dn/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/dn.recon.keytab. om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) kdc_1 | Oct 07 08:08:42 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43622:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:42 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) kdc_1 | Oct 07 08:08:42 kdc kadmind[15](Notice): Request: kadm5_create_principal, dn/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081547Z s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) kdc_1 | Oct 07 08:08:42 kdc kadmind[15](info): closing down fd 18 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, dn/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, dn/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Generiting keytab s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | WARNING: no policy specified for om/recon@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Principal "om/recon@EXAMPLE.COM" created. s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Entry for principal om/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.recon.keytab. s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Entry for principal om/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/om.recon.keytab. s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:42 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:08:42 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435722, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:42 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:42 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435722, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Generiting keytab om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | WARNING: no policy specified for scm/recon@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) kdc_1 | Principal "scm/recon@EXAMPLE.COM" created. s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Entry for principal scm/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.recon.keytab. s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Entry for principal scm/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/scm.recon.keytab. s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) kdc_1 | Generiting keytab kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. kdc_1 | WARNING: no policy specified for HTTP/recon@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Principal "HTTP/recon@EXAMPLE.COM" created. s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43624:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 20191007T081547Z kdc_1 | Entry for principal HTTP/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.recon.keytab. s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Entry for principal HTTP/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/HTTP.recon.keytab. s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) kdc_1 | Generiting keytab s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 20191007T081547Z kdc_1 | WARNING: no policy specified for testuser/recon@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Principal "testuser/recon@EXAMPLE.COM" created. s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) kdc_1 | Entry for principal testuser/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.recon.keytab. s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Entry for principal testuser/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser.recon.keytab. s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Generiting keytab s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | WARNING: no policy specified for testuser2/recon@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Principal "testuser2/recon@EXAMPLE.COM" created. s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Entry for principal testuser2/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.recon.keytab. s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Entry for principal testuser2/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/testuser2.recon.keytab. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Generiting keytab om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | WARNING: no policy specified for s3g/recon@EXAMPLE.COM; defaulting to no policy s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) kdc_1 | Principal "s3g/recon@EXAMPLE.COM" created. om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) kdc_1 | Authenticating as principal admin/admin with keytab /tmp/admin.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) kdc_1 | Entry for principal s3g/recon@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.recon.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) kdc_1 | Entry for principal s3g/recon@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/s3g.recon.keytab. om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_create_principal, om/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | ... 92 more kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, om/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43626:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, om/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | 20191007T081547Z om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_create_principal, scm/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, scm/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, scm/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_create_principal, HTTP/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, HTTP/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, HTTP/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43628:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_create_principal, testuser2/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | ... 101 more om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) s3g_1 | kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 om_1 | 2019-10-07 08:15:47 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, testuser2/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007T081547Z s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, testuser2/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_create_principal, s3g/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_init, admin/admin@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1, vers=4, flavor=6 s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_randkey_principal, s3g/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](Notice): Request: kadm5_get_principal, s3g/recon@EXAMPLE.COM, success, client=admin/admin@EXAMPLE.COM, service=kadmin/admin@EXAMPLE.COM, addr=127.0.0.1 s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:43 kdc kadmind[15](info): closing down fd 18 s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 2019-10-07 08:15:47 WARN Server:1833 - Auth failed for 172.18.0.9:43630:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 20191007T081547Z kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 20191007T081548Z s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: SERVER_NOT_FOUND: admin/admin@EXAMPLE.COM for kadmin/localhost@EXAMPLE.COM, Server not found in Kerberos database s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:08:43 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1570435723, etypes {rep=18 tkt=18 ses=18}, admin/admin@EXAMPLE.COM for kadmin/admin@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:08:49 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435729, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) kdc_1 | Oct 07 08:08:51 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435731, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:08:51 kdc krb5kdc[9](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435731, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for HTTP/scm@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:08:53 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435731, etypes {rep=18 tkt=18 ses=18}, HTTP/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:09:05 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:09:07 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:09:10 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:09:13 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) kdc_1 | Oct 07 08:09:16 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) kdc_1 | Oct 07 08:09:19 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) kdc_1 | Oct 07 08:09:21 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:09:24 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:09:27 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:09:30 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:09:35 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) kdc_1 | Oct 07 08:09:40 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) kdc_1 | Oct 07 08:09:43 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43634:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) kdc_1 | Oct 07 08:09:46 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) kdc_1 | Oct 07 08:09:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) kdc_1 | Oct 07 08:09:51 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) kdc_1 | Oct 07 08:09:54 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:09:57 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435745, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:09:58 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:10:00 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:10:03 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:10:06 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:10:09 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) kdc_1 | Oct 07 08:10:11 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:10:14 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:10:17 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:10:20 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435798, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:10:21 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:10:23 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) kdc_1 | Oct 07 08:10:26 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | 2019-10-07 08:15:47 WARN ServletHandler:625 - kdc_1 | Oct 07 08:10:29 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | javax.servlet.ServletException: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:10:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:10:34 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:10:37 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:10:40 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:10:43 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435821, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) kdc_1 | Oct 07 08:10:44 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) kdc_1 | Oct 07 08:10:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:10:52 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43636:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:10:55 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:10:58 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:11:01 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) kdc_1 | Oct 07 08:11:04 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:11:07 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:11:09 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435844, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) kdc_1 | Oct 07 08:11:10 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) kdc_1 | Oct 07 08:11:12 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:11:15 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:11:18 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:11:20 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:11:23 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) kdc_1 | Oct 07 08:11:26 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) kdc_1 | Oct 07 08:11:29 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) kdc_1 | Oct 07 08:11:32 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) kdc_1 | Oct 07 08:11:35 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) kdc_1 | Oct 07 08:11:40 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) kdc_1 | Oct 07 08:11:45 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:11:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:11:51 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) kdc_1 | Oct 07 08:11:53 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) kdc_1 | Oct 07 08:11:56 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) kdc_1 | Oct 07 08:11:59 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) kdc_1 | Oct 07 08:12:02 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435870, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) kdc_1 | Oct 07 08:12:03 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) kdc_1 | Oct 07 08:12:05 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:12:07 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:12:10 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43638:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:12:13 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:12:16 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) kdc_1 | Oct 07 08:12:19 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) kdc_1 | Oct 07 08:12:22 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:12:25 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 1. javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:12:27 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | kdc_1 | Oct 07 08:12:32 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:12:37 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) kdc_1 | Oct 07 08:12:40 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) kdc_1 | Oct 07 08:12:42 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) kdc_1 | Oct 07 08:12:45 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) kdc_1 | Oct 07 08:12:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:12:51 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:12:54 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435923, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:12:59 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:13:01 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:13:04 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:13:07 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:13:10 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:13:13 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) kdc_1 | Oct 07 08:13:16 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:13:18 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:13:21 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:13:24 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:13:27 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) kdc_1 | Oct 07 08:13:30 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) kdc_1 | Oct 07 08:13:33 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:13:36 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) kdc_1 | Oct 07 08:13:39 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) kdc_1 | Oct 07 08:13:42 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) kdc_1 | Oct 07 08:13:45 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43640:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) kdc_1 | Oct 07 08:13:47 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) kdc_1 | Oct 07 08:13:50 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | ... 33 more kdc_1 | Oct 07 08:13:53 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | Caused by: javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:13:56 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:13:59 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:14:01 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) kdc_1 | Oct 07 08:14:04 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) kdc_1 | Oct 07 08:14:09 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) kdc_1 | Oct 07 08:14:12 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) kdc_1 | Oct 07 08:14:15 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) kdc_1 | Oct 07 08:14:17 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) kdc_1 | Oct 07 08:14:20 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) kdc_1 | Oct 07 08:14:23 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:14:26 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) kdc_1 | Oct 07 08:14:29 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570435979, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) kdc_1 | Oct 07 08:14:30 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436070, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | Oct 07 08:14:32 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436070, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) kdc_1 | Oct 07 08:14:34 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436070, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) kdc_1 | Oct 07 08:14:37 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436070, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) kdc_1 | Oct 07 08:14:38 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436078, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) kdc_1 | Oct 07 08:14:40 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436078, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) kdc_1 | Oct 07 08:14:41 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436081, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:14:43 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436081, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:14:46 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436081, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) kdc_1 | Oct 07 08:14:49 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436081, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43642:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) kdc_1 | Oct 07 08:14:51 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436081, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) kdc_1 | Oct 07 08:14:54 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436081, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) kdc_1 | Oct 07 08:14:55 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436095, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) kdc_1 | Oct 07 08:14:57 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436095, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) kdc_1 | Oct 07 08:15:00 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436095, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) kdc_1 | Oct 07 08:15:01 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436101, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) kdc_1 | Oct 07 08:15:03 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436101, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) kdc_1 | Oct 07 08:15:06 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436101, etypes {rep=18 tkt=18 ses=18}, testuser2/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:15:07 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436107, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:15:37 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:15:39 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:16:02 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:16:05 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:16:08 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) kdc_1 | Oct 07 08:16:11 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) kdc_1 | Oct 07 08:16:14 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:16:17 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | ... 60 more om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:16:20 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:16:23 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:16:26 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:16:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) kdc_1 | Oct 07 08:16:34 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:16:38 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) kdc_1 | Oct 07 08:16:42 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) kdc_1 | Oct 07 08:16:45 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) kdc_1 | Oct 07 08:16:47 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:16:50 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:16:53 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:16:58 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:17:01 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | ... 92 more om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43644:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:17:04 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:17:09 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 20191007T081547Z kdc_1 | Oct 07 08:17:12 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:17:15 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:17:18 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:17:21 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:17:24 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:17:27 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) kdc_1 | Oct 07 08:17:30 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) kdc_1 | Oct 07 08:17:33 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) kdc_1 | Oct 07 08:17:36 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) kdc_1 | Oct 07 08:17:41 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:17:45 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:17:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) kdc_1 | Oct 07 08:17:51 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) kdc_1 | Oct 07 08:17:54 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) kdc_1 | Oct 07 08:17:59 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) kdc_1 | Oct 07 08:18:04 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:18:09 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:18:12 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:18:15 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:18:19 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.7: ISSUE: authtime 1570436137, etypes {rep=18 tkt=18 ses=18}, testuser/scm@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:18:51 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436331, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) kdc_1 | Oct 07 08:18:52 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436331, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) kdc_1 | Oct 07 08:18:57 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436337, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) kdc_1 | Oct 07 08:18:59 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436337, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:19:08 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436348, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:19:10 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436348, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43646:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | ... 101 more om_1 | 20191007T081548Z s3g_1 | 2019-10-07 08:15:47 WARN HttpChannel:499 - //s3g:9878/bucket-test123 om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:19:15 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436355, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:19:17 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436355, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | 1. javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:19:22 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436362, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | kdc_1 | Oct 07 08:19:24 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436362, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007T081548Z kdc_1 | Oct 07 08:19:29 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436369, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:19:31 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436369, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:19:39 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436379, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) kdc_1 | Oct 07 08:19:41 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436379, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) kdc_1 | Oct 07 08:19:49 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436389, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:19:51 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436389, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:19:58 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436398, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) kdc_1 | Oct 07 08:20:00 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436398, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) kdc_1 | Oct 07 08:20:06 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436406, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:20:08 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436406, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:20:13 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436413, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) kdc_1 | Oct 07 08:20:15 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436413, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) kdc_1 | Oct 07 08:20:20 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436420, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:20:22 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436420, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:20:26 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436426, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | Caused by: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:20:28 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436426, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | 1. javax.enterprise.inject.CreationException kdc_1 | Oct 07 08:20:37 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436437, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | kdc_1 | Oct 07 08:20:39 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436437, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) kdc_1 | Oct 07 08:20:46 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436446, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) kdc_1 | Oct 07 08:20:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436446, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) kdc_1 | Oct 07 08:20:56 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436456, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) kdc_1 | Oct 07 08:20:58 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436456, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:21:07 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436467, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43648:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:21:09 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436467, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:21:15 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436475, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) kdc_1 | Oct 07 08:21:17 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436475, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) kdc_1 | Oct 07 08:21:27 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436487, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) kdc_1 | Oct 07 08:21:29 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436487, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) kdc_1 | Oct 07 08:21:34 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436494, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 20191007T081548Z s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) kdc_1 | Oct 07 08:21:35 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436494, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) kdc_1 | Oct 07 08:21:40 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436500, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) kdc_1 | Oct 07 08:21:41 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436500, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:21:46 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436506, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) kdc_1 | Oct 07 08:21:48 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436506, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) kdc_1 | Oct 07 08:21:54 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436514, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) kdc_1 | Oct 07 08:21:56 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436514, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) kdc_1 | Oct 07 08:22:01 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436521, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:22:03 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436521, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) kdc_1 | Oct 07 08:22:10 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436530, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | ... 13 more kdc_1 | Oct 07 08:22:12 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436530, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: kdc_1 | Oct 07 08:22:19 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436539, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:22:21 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436539, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) kdc_1 | Oct 07 08:22:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436545, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) kdc_1 | Oct 07 08:22:27 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436545, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) kdc_1 | Oct 07 08:22:36 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436556, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) kdc_1 | Oct 07 08:22:38 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436556, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) kdc_1 | Oct 07 08:22:45 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436565, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) kdc_1 | Oct 07 08:22:47 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436565, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43650:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:22:51 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436571, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 20191007T081548Z kdc_1 | Oct 07 08:22:53 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436571, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) kdc_1 | Oct 07 08:22:59 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436579, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 kdc_1 | Oct 07 08:23:01 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436579, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | 20191007T081548Z kdc_1 | Oct 07 08:23:05 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436585, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 20191007/us-west-1/s3/aws4_request kdc_1 | Oct 07 08:23:07 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436585, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf kdc_1 | Oct 07 08:23:11 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436591, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf kdc_1 | Oct 07 08:23:13 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436591, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) kdc_1 | Oct 07 08:23:18 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436598, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) kdc_1 | Oct 07 08:23:20 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436598, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) kdc_1 | Oct 07 08:23:25 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436605, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) kdc_1 | Oct 07 08:23:27 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436605, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) kdc_1 | Oct 07 08:23:32 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436612, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) kdc_1 | Oct 07 08:23:34 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436612, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) kdc_1 | Oct 07 08:23:39 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436619, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) kdc_1 | Oct 07 08:23:41 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436619, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) kdc_1 | Oct 07 08:23:46 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436626, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) kdc_1 | Oct 07 08:23:49 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436626, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) kdc_1 | Oct 07 08:23:53 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436633, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) kdc_1 | Oct 07 08:23:55 kdc krb5kdc[9](info): TGS_REQ (6 etypes {18 17 20 19 16 23}) 172.18.0.9: ISSUE: authtime 1570436633, etypes {rep=18 tkt=18 ses=18}, testuser/s3g@EXAMPLE.COM for om/om@EXAMPLE.COM s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) kdc_1 | Oct 07 08:23:57 kdc krb5kdc[9](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436637, etypes {rep=18 tkt=18 ses=18}, HTTP/s3g@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) kdc_1 | Oct 07 08:23:57 kdc krb5kdc[9](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 172.18.0.9: ISSUE: authtime 1570436637, etypes {rep=18 tkt=18 ses=18}, HTTP/s3g@EXAMPLE.COM for HTTP/s3g@EXAMPLE.COM om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | ... 33 more om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43652:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | Caused by: javax.enterprise.inject.CreationException om_1 | 20191007T081548Z s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) om_1 | 2019-10-07 08:15:49 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) om_1 | 20191007T081548Z s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | 20191007/us-west-1/s3/aws4_request om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | 2019-10-07 08:15:49 WARN Server:1833 - Auth failed for 172.18.0.9:43654:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | 20191007T081548Z s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | 20191007/us-west-1/s3/aws4_request om_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | 20191007T081552Z s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | ... 60 more s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | 20191007T081547Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 585ae503338274dc86f9e28adf8db729bf58ac2ce3e75678b16dfb76d28322e5, signature=87dc6032fef45d86a3d2108c098cf4fb19e81f58cc323f0a4cbb858a0b5357f4, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43662:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081552Z s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081552Z s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 20191007/us-west-1/s3/aws4_request om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43664:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081552Z s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | ... 101 more om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z om_1 | 20191007T081552Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 2 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 3 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43666:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z om_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 20191007/us-west-1/s3/aws4_request om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | 20191007T081552Z s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 4 failover attempts. Trying to failover immediately. om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | 20191007T081548Z om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 5 failover attempts. Trying to failover immediately. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 6 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43668:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | 20191007T081552Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | 20191007T081548Z om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 20191007T081552Z s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 7 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 20191007T081548Z om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 8 failover attempts. Trying to failover immediately. om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | 2019-10-07 08:15:49 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 9 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | 2019-10-07 08:15:49 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43670:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | 20191007T081552Z s3g_1 | 2019-10-07 08:15:49 ERROR OMFailoverProxyProvider:268 - Failed to connect to OM. Attempted 10 retries and 10 failovers om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 2019-10-07 08:15:49 ERROR OzoneClientFactory:259 - Couldn't create RpcClient protocol exception: om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z om_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43672:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081552Z om_1 | 20191007/us-west-1/s3/aws4_request om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | 20191007T081552Z s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43674:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | 20191007T081552Z s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | 20191007T081552Z s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43676:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 20191007T081552Z s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 20191007/us-west-1/s3/aws4_request om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081552Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43678:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | 20191007T081552Z s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 20191007T081552Z s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43680:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 20191007T081552Z s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:15:52 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 20191007T081552Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | Oct 07, 2019 8:15:49 AM org.glassfish.jersey.internal.Errors logErrors s3g_1 | WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected: om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | MultiException stack 1 of 1 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | 2019-10-07 08:15:52 WARN Server:1833 - Auth failed for 172.18.0.9:43682:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081552Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 2019-10-07 08:15:58 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 20191007T081558Z s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:58 WARN Server:1833 - Auth failed for 172.18.0.9:43696:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | 20191007T081558Z s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | 2019-10-07 08:15:58 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:58 WARN Server:1833 - Auth failed for 172.18.0.9:43698:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | 2019-10-07 08:15:58 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:15:58 WARN Server:1833 - Auth failed for 172.18.0.9:43700:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 20191007T081558Z s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 2019-10-07 08:15:58 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081558Z s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | 2019-10-07 08:15:58 WARN Server:1833 - Auth failed for 172.18.0.9:43702:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081558Z om_1 | 20191007/us-west-1/s3/aws4_request om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) om_1 | 2019-10-07 08:15:58 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 20191007T081558Z s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | ... 92 more om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | 20191007T081548Z om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:15:58 WARN Server:1833 - Auth failed for 172.18.0.9:43704:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | 20191007T081558Z s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | 2019-10-07 08:15:58 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | 20191007T081558Z s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | ... 101 more om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | om_1 | 2019-10-07 08:15:58 WARN Server:1833 - Auth failed for 172.18.0.9:43706:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | om_1 | 20191007T081558Z s3g_1 | 2019-10-07 08:15:49 WARN ServletHandler:625 - om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | javax.servlet.ServletException: A MultiException has 1 exceptions. They are: om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:15:59 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-07 08:15:59 WARN Server:1833 - Auth failed for 172.18.0.9:43708:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 20191007T081558Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) om_1 | 2019-10-07 08:15:59 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081558Z s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 2019-10-07 08:15:59 WARN Server:1833 - Auth failed for 172.18.0.9:43710:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:15:59 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | ... 33 more om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | Caused by: javax.enterprise.inject.CreationException s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | 2019-10-07 08:15:59 WARN Server:1833 - Auth failed for 172.18.0.9:43712:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081558Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | 2019-10-07 08:15:59 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | 20191007T081558Z s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | 20191007/us-west-1/s3/aws4_request om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) om_1 | 2019-10-07 08:15:59 WARN Server:1833 - Auth failed for 172.18.0.9:43714:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 20191007T081558Z om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 2019-10-07 08:15:59 ERROR OzoneDelegationTokenSecretManager:402 - Error while validating S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | 20191007T081558Z s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | ... 60 more om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | org.apache.hadoop.ozone.security.OzoneSecurityException: S3 secret not found for awsAccessKeyId dlfknslnfslf s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | at org.apache.hadoop.ozone.om.S3SecretManagerImpl.getS3UserSecretString(S3SecretManagerImpl.java:94) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.validateS3Token(OzoneDelegationTokenSecretManager.java:399) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:347) om_1 | at org.apache.hadoop.ozone.security.OzoneDelegationTokenSecretManager.retrievePassword(OzoneDelegationTokenSecretManager.java:56) om_1 | at org.apache.hadoop.security.token.SecretManager.retriableRetrievePassword(SecretManager.java:91) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:277) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:304) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | at java.security.sasl/com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslToken(Server.java:1955) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ipc.Server$Connection.processSaslMessage(Server.java:1932) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslProcess(Server.java:1825) om_1 | at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1767) om_1 | at org.apache.hadoop.ipc.Server$Connection.processRpcOutOfBandRequest(Server.java:2532) om_1 | at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:2360) s3g_1 | ... 92 more om_1 | at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:2109) s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:1249) s3g_1 | 20191007T081548Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:1105) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:1076) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | 2019-10-07 08:15:59 WARN Server:1833 - Auth failed for 172.18.0.9:43716:null (DIGEST-MD5: IO error acquiring password) with true cause: (No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 20191007T081558Z s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 20191007/us-west-1/s3/aws4_request om_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf) om_1 | 2019-10-07 08:16:02 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:16:02 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | 2019-10-07 08:16:03 INFO OMVolumeCreateRequest:193 - created volume:fstest for user:testuser/scm@EXAMPLE.COM s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | 2019-10-07 08:16:05 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | 2019-10-07 08:16:05 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | 2019-10-07 08:16:06 INFO OMVolumeCreateRequest:193 - created volume:fstest2 for user:testuser/scm@EXAMPLE.COM s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | 2019-10-07 08:16:08 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | 2019-10-07 08:16:08 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | 2019-10-07 08:16:11 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | 2019-10-07 08:16:11 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 2019-10-07 08:16:14 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | 2019-10-07 08:16:14 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | 2019-10-07 08:16:17 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | 2019-10-07 08:16:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | ... 101 more om_1 | 2019-10-07 08:16:20 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:16:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:16:23 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 2019-10-07 08:15:49 WARN HttpChannel:499 - //s3g:9878/bucket-test123 om_1 | 2019-10-07 08:16:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: om_1 | 2019-10-07 08:16:26 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | om_1 | 2019-10-07 08:16:26 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) om_1 | 2019-10-07 08:16:31 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 2019-10-07 08:16:31 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:16:34 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:16:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-07 08:16:38 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 2019-10-07 08:16:38 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:16:42 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 2019-10-07 08:16:42 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | 2019-10-07 08:16:45 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:16:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 2019-10-07 08:16:47 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | 2019-10-07 08:16:47 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:16:50 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Caused by: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | om_1 | 2019-10-07 08:16:50 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) om_1 | 2019-10-07 08:16:53 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:16:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:16:58 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:16:58 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 2019-10-07 08:17:01 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 2019-10-07 08:17:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 2019-10-07 08:17:04 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:17:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:17:09 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:17:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 2019-10-07 08:17:12 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 2019-10-07 08:17:12 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | 2019-10-07 08:17:15 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:17:15 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:17:18 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | 2019-10-07 08:17:18 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | 2019-10-07 08:17:21 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | 2019-10-07 08:17:21 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | 2019-10-07 08:17:24 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | 2019-10-07 08:17:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | 2019-10-07 08:17:27 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:17:27 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:17:30 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 2019-10-07 08:17:30 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | ... 13 more om_1 | 2019-10-07 08:17:33 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: om_1 | 2019-10-07 08:17:33 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:17:36 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:17:36 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) om_1 | 2019-10-07 08:17:41 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | 2019-10-07 08:17:41 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | 2019-10-07 08:17:45 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | 2019-10-07 08:17:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | 2019-10-07 08:17:48 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 2019-10-07 08:17:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 2019-10-07 08:17:51 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | 2019-10-07 08:17:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:17:55 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | 2019-10-07 08:17:55 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:17:59 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:17:59 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | 2019-10-07 08:18:04 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | 2019-10-07 08:18:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:18:09 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:18:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 2019-10-07 08:18:12 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | 2019-10-07 08:18:12 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:18:15 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | 2019-10-07 08:18:15 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 2019-10-07 08:18:19 INFO Server:1849 - Auth successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | 2019-10-07 08:18:19 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/scm@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | 2019-10-07 08:18:20 ERROR VolumeManagerImpl:692 - Check access operation failed for volume:pqrs om_1 | VOLUME_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Volume pqrs is not found s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | at org.apache.hadoop.ozone.om.VolumeManagerImpl.checkAccess(VolumeManagerImpl.java:681) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | at org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer.checkAccess(OzoneNativeAuthorizer.java:83) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | ... 33 more s3g_1 | Caused by: javax.enterprise.inject.CreationException om_1 | at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1620) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1586) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | at org.apache.hadoop.ozone.om.OzoneManager.getVolumeInfo(OzoneManager.java:1743) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.infoVolume(OzoneManagerRequestHandler.java:518) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handle(OzoneManagerRequestHandler.java:175) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:210) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | 2019-10-07 08:18:53 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | 2019-10-07 08:18:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | 2019-10-07 08:18:56 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | 2019-10-07 08:18:56 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:18:56 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | 2019-10-07 08:18:59 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:18:59 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 2019-10-07 08:19:02 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | 2019-10-07 08:19:02 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | 2019-10-07 08:19:02 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | 2019-10-07 08:19:05 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | 2019-10-07 08:19:05 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | 2019-10-07 08:19:05 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:06 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:19:06 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 2019-10-07 08:19:06 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | 2019-10-07 08:19:07 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 2019-10-07 08:19:07 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) om_1 | 2019-10-07 08:19:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) om_1 | 2019-10-07 08:19:07 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | 2019-10-07 08:19:07 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:19:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:10 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:19:10 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:13 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | ... 60 more om_1 | 2019-10-07 08:19:13 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | 2019-10-07 08:19:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | 2019-10-07 08:19:14 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | 2019-10-07 08:19:14 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | 2019-10-07 08:19:14 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 2019-10-07 08:19:14 ERROR OzoneManagerProtocolProtos$S3CreateBucketRequest:252 - S3Bucket Creation Failed for userName: 2724f42bcd3225359401cb62da89c51d, s3BucketName bucket-53857, VolumeName s32724f42bcd3225359401cb62da89c51d om_1 | 2019-10-07 08:19:17 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:19:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081548Z s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:20 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 4a82ed839519a1b7bf39ab1da8cf0d97ee6d296bdc50768c4328909ee94aa239, signature=5c302b26727cb1c6e085a649602bb740b0f07f58a57cfb3556b140cd4e309b97, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:19:20 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | 2019-10-07 08:19:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:21 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | 2019-10-07 08:19:21 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | 2019-10-07 08:19:21 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) om_1 | 2019-10-07 08:19:22 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:19:22 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | 2019-10-07 08:19:22 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | 2019-10-07 08:19:24 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:19:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | 2019-10-07 08:19:28 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:19:28 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | 2019-10-07 08:19:28 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | 2019-10-07 08:19:28 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 2019-10-07 08:19:28 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:19:28 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | 2019-10-07 08:19:31 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:19:31 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | 2019-10-07 08:19:35 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | 2019-10-07 08:19:35 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | ... 101 more s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:35 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:35 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 2 failover attempts. Trying to failover immediately. om_1 | 2019-10-07 08:19:36 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:36 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:36 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:37 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:19:37 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:37 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:38 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:38 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:38 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:41 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 3 failover attempts. Trying to failover immediately. om_1 | 2019-10-07 08:19:41 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:44 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:44 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:19:44 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:45 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:45 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:19:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:46 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:46 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 4 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:46 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:19:47 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:47 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:19:47 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:48 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 5 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:48 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:19:51 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:54 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:54 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:19:54 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:55 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:55 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:19:55 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:56 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 6 failover attempts. Trying to failover immediately. om_1 | 2019-10-07 08:19:56 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:19:56 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:19:57 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:19:57 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:19:57 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:19:57 ERROR S3MultipartUploadCompleteRequest:205 - MultipartUpload: /s32724f42bcd3225359401cb62da89c51d/bucket-85245/multipartKey2Part number: 1size 6 is less than minimum part size 5242880 s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:19:57 ERROR S3MultipartUploadCompleteRequest:293 - MultipartUpload Complete request failed for Key: multipartKey2 in Volume/Bucket s32724f42bcd3225359401cb62da89c51d/bucket-85245 s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | ENTITY_TOO_SMALL org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: Entity too small: volume: s32724f42bcd3225359401cb62da89c51dbucket: bucket-85245key: multipartKey2 s3g_1 | 20191007T081552Z om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:209) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 7 failover attempts. Trying to failover immediately. om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) s3g_1 | 20191007T081552Z om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) om_1 | 2019-10-07 08:20:00 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:20:00 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:20:03 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 8 failover attempts. Trying to failover immediately. om_1 | 2019-10-07 08:20:03 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:20:03 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:20:04 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:20:04 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:20:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 2019-10-07 08:15:52 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | 2019-10-07 08:20:05 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:20:05 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | 2019-10-07 08:20:05 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 9 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:52 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:20:06 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:20:06 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:06 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 2019-10-07 08:15:52 ERROR OMFailoverProxyProvider:268 - Failed to connect to OM. Attempted 10 retries and 10 failovers om_1 | 2019-10-07 08:20:06 ERROR S3MultipartUploadCompleteRequest:293 - MultipartUpload Complete request failed for Key: multipartKey3 in Volume/Bucket s32724f42bcd3225359401cb62da89c51d/bucket-85245 s3g_1 | 2019-10-07 08:15:52 ERROR OzoneClientFactory:259 - Couldn't create RpcClient protocol exception: om_1 | MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: volume: s32724f42bcd3225359401cb62da89c51dbucket: bucket-85245key: multipartKey3 s3g_1 | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:195) s3g_1 | 20191007T081552Z om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) s3g_1 | 20191007/us-west-1/s3/aws4_request om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) om_1 | 2019-10-07 08:20:08 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:20:08 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:12 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:20:12 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | 2019-10-07 08:20:12 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 2019-10-07 08:20:13 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | 2019-10-07 08:20:13 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) om_1 | 2019-10-07 08:20:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) om_1 | 2019-10-07 08:20:15 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | 2019-10-07 08:20:15 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 2019-10-07 08:20:19 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | 2019-10-07 08:20:19 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | 2019-10-07 08:20:19 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | 2019-10-07 08:20:19 ERROR S3MultipartUploadAbortRequest:166 - Abort Multipart request is failed for KeyName multipartKey5 in VolumeName/Bucket s32724f42bcd3225359401cb62da89c51d/bucket-85245 s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Abort Multipart Upload Failed: volume: s32724f42bcd3225359401cb62da89c51dbucket: bucket-85245key: multipartKey5 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:116) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | 2019-10-07 08:20:22 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | 2019-10-07 08:20:22 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 2019-10-07 08:20:25 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 2019-10-07 08:20:25 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | 2019-10-07 08:20:25 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | 2019-10-07 08:20:25 ERROR OMKeyRequest:330 - ALLOCATE_KEY failed for Key: multipartKey in volume/bucket:s32724f42bcd3225359401cb62da89c51d/bucket-85245 s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload is with specified uploadId random s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareMultipartKeyInfo(OMKeyRequest.java:471) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareKeyInfo(OMKeyRequest.java:423) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:179) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | 2019-10-07 08:20:28 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | 2019-10-07 08:20:28 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 2019-10-07 08:20:31 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | 2019-10-07 08:20:31 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) om_1 | 2019-10-07 08:20:31 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) om_1 | 2019-10-07 08:20:32 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) om_1 | 2019-10-07 08:20:32 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) om_1 | 2019-10-07 08:20:32 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | 2019-10-07 08:20:33 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | 2019-10-07 08:20:33 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) om_1 | 2019-10-07 08:20:33 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | 2019-10-07 08:20:34 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 2019-10-07 08:20:34 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:35 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:20:35 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 2019-10-07 08:20:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | 2019-10-07 08:20:35 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | 2019-10-07 08:20:35 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:20:36 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:20:36 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:20:36 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 2019-10-07 08:20:39 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 2019-10-07 08:20:39 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:42 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) om_1 | 2019-10-07 08:20:42 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 2019-10-07 08:20:42 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | 2019-10-07 08:20:42 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | 2019-10-07 08:20:42 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | 2019-10-07 08:20:42 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:20:42 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | 2019-10-07 08:20:42 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | 2019-10-07 08:20:43 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 2019-10-07 08:20:43 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | 2019-10-07 08:20:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | 2019-10-07 08:20:43 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 2019-10-07 08:20:43 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 2019-10-07 08:20:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 2019-10-07 08:20:44 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:20:44 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:44 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 2019-10-07 08:20:44 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 2019-10-07 08:20:44 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:20:44 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:20:45 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | 2019-10-07 08:20:45 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:48 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) om_1 | 2019-10-07 08:20:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | 2019-10-07 08:20:51 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | 2019-10-07 08:20:51 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | 2019-10-07 08:20:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:52 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | 2019-10-07 08:20:52 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:52 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:20:53 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | 2019-10-07 08:20:53 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:20:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:20:54 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 2019-10-07 08:20:54 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 2019-10-07 08:20:54 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 2019-10-07 08:20:55 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 2019-10-07 08:20:55 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:20:55 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | 2019-10-07 08:20:58 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:20:58 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 2019-10-07 08:21:01 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | 2019-10-07 08:21:01 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | 2019-10-07 08:21:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | 2019-10-07 08:21:02 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | 2019-10-07 08:21:02 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | 2019-10-07 08:21:02 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | Oct 07, 2019 8:15:52 AM org.glassfish.jersey.internal.Errors logErrors om_1 | 2019-10-07 08:21:03 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected: om_1 | 2019-10-07 08:21:03 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | MultiException stack 1 of 1 s3g_1 | javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:21:03 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | 2019-10-07 08:21:04 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | 2019-10-07 08:21:04 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | 2019-10-07 08:21:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | 2019-10-07 08:21:05 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:21:05 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | 2019-10-07 08:21:05 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | 2019-10-07 08:21:06 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:21:06 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) om_1 | 2019-10-07 08:21:06 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | 2019-10-07 08:21:09 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | 2019-10-07 08:21:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | 2019-10-07 08:21:13 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | 2019-10-07 08:21:13 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 2019-10-07 08:21:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | 2019-10-07 08:21:13 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | 2019-10-07 08:21:13 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) om_1 | 2019-10-07 08:21:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) om_1 | 2019-10-07 08:21:14 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) om_1 | 2019-10-07 08:21:14 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) om_1 | 2019-10-07 08:21:14 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) om_1 | 2019-10-07 08:21:17 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:21:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) om_1 | 2019-10-07 08:21:20 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) om_1 | 2019-10-07 08:21:20 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) om_1 | 2019-10-07 08:21:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 2019-10-07 08:21:21 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) om_1 | 2019-10-07 08:21:21 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:21 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) om_1 | 2019-10-07 08:21:22 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) om_1 | 2019-10-07 08:21:22 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) om_1 | 2019-10-07 08:21:22 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) om_1 | 2019-10-07 08:21:23 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) om_1 | 2019-10-07 08:21:23 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) om_1 | 2019-10-07 08:21:23 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:21:23 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) om_1 | 2019-10-07 08:21:24 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) om_1 | 2019-10-07 08:21:24 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) om_1 | 2019-10-07 08:21:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) om_1 | 2019-10-07 08:21:25 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) om_1 | 2019-10-07 08:21:25 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | 2019-10-07 08:21:25 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | 2019-10-07 08:21:26 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | 2019-10-07 08:21:26 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:26 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 2019-10-07 08:21:29 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 2019-10-07 08:21:29 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | 2019-10-07 08:21:32 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:21:32 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | 2019-10-07 08:21:32 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) om_1 | 2019-10-07 08:21:33 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | 2019-10-07 08:21:33 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:33 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:21:35 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:21:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 2019-10-07 08:21:39 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 2019-10-07 08:21:39 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) om_1 | 2019-10-07 08:21:39 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 2019-10-07 08:21:41 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | 2019-10-07 08:21:41 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | 2019-10-07 08:21:45 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) om_1 | 2019-10-07 08:21:45 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 2019-10-07 08:21:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 2019-10-07 08:21:48 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 2019-10-07 08:21:48 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 2019-10-07 08:21:51 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:21:51 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 2019-10-07 08:21:51 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | 2019-10-07 08:21:51 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:21:51 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:21:52 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:21:52 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | 2019-10-07 08:21:52 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | 2019-10-07 08:21:53 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | 2019-10-07 08:21:53 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | 2019-10-07 08:21:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:21:54 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:21:54 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:21:54 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 2019-10-07 08:21:56 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 2019-10-07 08:21:56 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-07 08:21:59 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 2019-10-07 08:21:59 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 2019-10-07 08:21:59 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | 2019-10-07 08:22:00 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:22:00 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:00 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:01 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:22:01 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 2019-10-07 08:22:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:03 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | 2019-10-07 08:22:03 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) om_1 | 2019-10-07 08:22:07 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | 2019-10-07 08:22:07 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) om_1 | 2019-10-07 08:22:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) om_1 | 2019-10-07 08:22:07 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | 2019-10-07 08:22:07 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol om_1 | 2019-10-07 08:22:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) om_1 | 2019-10-07 08:22:08 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) om_1 | 2019-10-07 08:22:08 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:08 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:09 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) om_1 | 2019-10-07 08:22:09 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) om_1 | 2019-10-07 08:22:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) om_1 | 2019-10-07 08:22:09 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | 2019-10-07 08:22:09 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | 2019-10-07 08:22:09 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 2019-10-07 08:22:12 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) om_1 | 2019-10-07 08:22:12 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z om_1 | 2019-10-07 08:22:16 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf om_1 | 2019-10-07 08:22:16 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) om_1 | 2019-10-07 08:22:16 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) om_1 | 2019-10-07 08:22:17 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:22:17 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) om_1 | 2019-10-07 08:22:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) om_1 | 2019-10-07 08:22:17 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:22:17 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) om_1 | 2019-10-07 08:22:18 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 2019-10-07 08:22:18 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) om_1 | 2019-10-07 08:22:18 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 2019-10-07 08:22:18 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:22:18 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) om_1 | 2019-10-07 08:22:18 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) om_1 | 2019-10-07 08:22:21 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) om_1 | 2019-10-07 08:22:21 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) om_1 | 2019-10-07 08:22:24 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) om_1 | 2019-10-07 08:22:24 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:27 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) om_1 | 2019-10-07 08:22:27 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) om_1 | 2019-10-07 08:22:30 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | 2019-10-07 08:22:30 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | 2019-10-07 08:22:30 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | 2019-10-07 08:22:31 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) om_1 | 2019-10-07 08:22:31 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:31 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) om_1 | 2019-10-07 08:22:32 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) om_1 | 2019-10-07 08:22:32 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) om_1 | 2019-10-07 08:22:32 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:33 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | ... 101 more om_1 | 2019-10-07 08:22:33 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | om_1 | 2019-10-07 08:22:33 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | om_1 | 2019-10-07 08:22:34 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | 2019-10-07 08:15:52 WARN ServletHandler:625 - om_1 | 2019-10-07 08:22:34 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | javax.servlet.ServletException: A MultiException has 1 exceptions. They are: om_1 | 2019-10-07 08:22:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:22:34 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | om_1 | 2019-10-07 08:22:34 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) om_1 | 2019-10-07 08:22:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | 2019-10-07 08:22:35 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 2019-10-07 08:22:35 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | 2019-10-07 08:22:35 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | 2019-10-07 08:22:38 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | 2019-10-07 08:22:38 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | 2019-10-07 08:22:41 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | 2019-10-07 08:22:41 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:41 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:42 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:22:42 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:42 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:22:43 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-07 08:22:43 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | 2019-10-07 08:22:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | 2019-10-07 08:22:43 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) om_1 | 2019-10-07 08:22:43 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) om_1 | 2019-10-07 08:22:43 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) om_1 | 2019-10-07 08:22:44 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) om_1 | 2019-10-07 08:22:44 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) om_1 | 2019-10-07 08:22:44 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-07 08:22:47 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-07 08:22:47 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 2019-10-07 08:22:50 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 2019-10-07 08:22:50 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-07 08:22:50 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 2019-10-07 08:22:53 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) om_1 | 2019-10-07 08:22:53 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) om_1 | 2019-10-07 08:22:56 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) om_1 | 2019-10-07 08:22:56 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:56 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:22:57 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:22:57 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:22:57 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) om_1 | 2019-10-07 08:22:58 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) om_1 | 2019-10-07 08:22:58 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) om_1 | 2019-10-07 08:22:58 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) om_1 | 2019-10-07 08:23:01 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: om_1 | 2019-10-07 08:23:01 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | 1. javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:23:04 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | om_1 | 2019-10-07 08:23:04 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) om_1 | 2019-10-07 08:23:04 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) om_1 | 2019-10-07 08:23:07 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) om_1 | 2019-10-07 08:23:07 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) om_1 | 2019-10-07 08:23:10 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:23:10 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:23:11 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:23:13 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) om_1 | 2019-10-07 08:23:13 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) om_1 | 2019-10-07 08:23:17 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) om_1 | 2019-10-07 08:23:17 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:23:17 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:23:20 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) om_1 | 2019-10-07 08:23:20 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) om_1 | 2019-10-07 08:23:23 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) om_1 | 2019-10-07 08:23:23 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:23:23 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:23:24 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:23:24 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:23:24 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) om_1 | 2019-10-07 08:23:27 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) om_1 | 2019-10-07 08:23:27 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) om_1 | 2019-10-07 08:23:30 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:23:30 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:23:30 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:23:31 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:23:31 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 2019-10-07 08:23:31 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | 2019-10-07 08:23:34 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) om_1 | 2019-10-07 08:23:34 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:23:37 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:23:37 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) om_1 | 2019-10-07 08:23:37 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol om_1 | 2019-10-07 08:23:38 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 om_1 | 2019-10-07 08:23:38 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | 2019-10-07 08:23:38 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | 2019-10-07 08:23:41 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | 2019-10-07 08:23:41 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | ... 33 more s3g_1 | Caused by: javax.enterprise.inject.CreationException om_1 | 2019-10-07 08:23:44 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) om_1 | 2019-10-07 08:23:44 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) om_1 | 2019-10-07 08:23:44 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) om_1 | 2019-10-07 08:23:45 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) om_1 | 2019-10-07 08:23:45 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) om_1 | 2019-10-07 08:23:45 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) om_1 | 2019-10-07 08:23:46 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) om_1 | 2019-10-07 08:23:46 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) om_1 | 2019-10-07 08:23:46 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) om_1 | 2019-10-07 08:23:49 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) om_1 | 2019-10-07 08:23:49 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) om_1 | 2019-10-07 08:23:52 INFO AWSV4AuthValidator:98 - 434a4dac0d7fd5a7a31854108a68c07778e82ce00c776fd2f6be553b99b67897 s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) om_1 | 2019-10-07 08:23:52 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) om_1 | 2019-10-07 08:23:52 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:TOKEN) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) om_1 | 2019-10-07 08:23:55 INFO Server:1849 - Auth successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) om_1 | 2019-10-07 08:23:55 INFO ServiceAuthorizationManager:138 - Authorization successful for testuser/s3g@EXAMPLE.COM (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | ... 60 more s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | ... 101 more s3g_1 | 2019-10-07 08:15:52 WARN HttpChannel:499 - //s3g:9878/bucket-test123 s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Caused by: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | ... 13 more s3g_1 | Caused by: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | ... 33 more s3g_1 | Caused by: javax.enterprise.inject.CreationException s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | ... 60 more s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081552Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | 62b799d09ff4264a282377c49c0dd3c176abdff262448a0090d26ca9e15abbff, signature=934ddaa951e53bbd7dce9c4d3109eda7363742acd1d519d406a67e6dab063ad2, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | ... 101 more s3g_1 | 2019-10-07 08:15:58 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:58 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:58 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:58 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:58 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 2 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:58 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:58 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 3 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:58 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:58 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 4 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:58 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:58 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 5 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:59 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:59 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 6 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:59 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:59 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 7 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:59 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:59 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 8 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:59 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:59 INFO RetryInvocationHandler:411 - com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf, while invoking $Proxy93.submitRequest over nodeId=null,nodeAddress=om:9862 after 9 failover attempts. Trying to failover immediately. s3g_1 | 2019-10-07 08:15:59 WARN Client:755 - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | 2019-10-07 08:15:59 ERROR OMFailoverProxyProvider:268 - Failed to connect to OM. Attempted 10 retries and 10 failovers s3g_1 | 2019-10-07 08:15:59 ERROR OzoneClientFactory:259 - Couldn't create RpcClient protocol exception: s3g_1 | org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Oct 07, 2019 8:15:59 AM org.glassfish.jersey.internal.Errors logErrors s3g_1 | WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected: s3g_1 | MultiException stack 1 of 1 s3g_1 | javax.enterprise.inject.CreationException s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | ... 101 more s3g_1 | s3g_1 | s3g_1 | 2019-10-07 08:15:59 WARN ServletHandler:625 - s3g_1 | javax.servlet.ServletException: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Caused by: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | ... 33 more s3g_1 | Caused by: javax.enterprise.inject.CreationException s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | ... 60 more s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | ... 101 more s3g_1 | 2019-10-07 08:15:59 WARN HttpChannel:499 - //s3g:9878/bucket-test123 s3g_1 | javax.servlet.ServletException: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:139) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Caused by: javax.servlet.ServletException: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | ... 13 more s3g_1 | Caused by: A MultiException has 1 exceptions. They are: s3g_1 | 1. javax.enterprise.inject.CreationException s3g_1 | s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:494) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | ... 33 more s3g_1 | Caused by: javax.enterprise.inject.CreationException s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) s3g_1 | at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) s3g_1 | at java.base/java.lang.Class.newInstance(Class.java:584) s3g_1 | at org.jboss.weld.security.NewInstanceAction.run(NewInstanceAction.java:33) s3g_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:40) s3g_1 | at org.jboss.weld.injection.Exceptions.rethrowException(Exceptions.java:78) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:96) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | ... 60 more s3g_1 | Caused by: java.io.IOException: Couldn't create RpcClient protocol s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:263) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:239) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:75) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:113) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | ... 92 more s3g_1 | Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): No S3 secret found for S3 identifier:OzoneToken owner=dlfknslnfslf, renewer=, realUser=, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0, strToSign=AWS4-HMAC-SHA256 s3g_1 | 20191007T081558Z s3g_1 | 20191007/us-west-1/s3/aws4_request s3g_1 | e38761aea01e723352bf5d9bbe420f0fc052e31d8db1a83269f7427c1ff9820a, signature=2b185e6ac3375bfdb2d9531208c0665c94084a0c24faf83ac83cec04ef8b4c17, awsAccessKeyId=dlfknslnfslf s3g_1 | at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1457) s3g_1 | at org.apache.hadoop.ipc.Client.call(Client.java:1367) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) s3g_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) s3g_1 | at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) s3g_1 | at com.sun.proxy.$Proxy93.submitRequest(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:338) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1223) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.getServiceInfo(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:256) s3g_1 | ... 101 more s3g_1 | 2019-10-07 08:18:56 INFO BucketEndpoint:211 - Location is /bucket-30918 s3g_1 | 2019-10-07 08:19:03 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties s3g_1 | 2019-10-07 08:19:03 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). s3g_1 | 2019-10-07 08:19:03 INFO MetricsSystemImpl:191 - XceiverClientMetrics metrics system started s3g_1 | 2019-10-07 08:19:03 WARN MetricsSystemImpl:276 - Sink prometheus already exists! s3g_1 | 2019-10-07 08:19:13 INFO BucketEndpoint:211 - Location is /bucket-53857 s3g_1 | 2019-10-07 08:19:14 INFO BucketEndpoint:211 - Location is /bucket-53857 s3g_1 | 2019-10-07 08:19:20 INFO BucketEndpoint:211 - Location is /bucket-95916 s3g_1 | 2019-10-07 08:19:22 ERROR BucketEndpoint:252 - Exception occurred in headBucket s3g_1 | org.apache.hadoop.ozone.s3.exception.OS3Exception s3g_1 | at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:103) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getBucket(EndpointBase.java:81) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.head(BucketEndpoint.java:250) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | 2019-10-07 08:19:28 INFO BucketEndpoint:211 - Location is /bucket-30581 s3g_1 | 2019-10-07 08:19:35 INFO BucketEndpoint:211 - Location is /bucket-85245 s3g_1 | 2019-10-07 08:19:57 ERROR ObjectEndpoint:511 - Error in Complete Multipart Upload Request for bucket: bucket-85245, key: multipartKey2 s3g_1 | ENTITY_TOO_SMALL org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: Entity too small: volume: s32724f42bcd3225359401cb62da89c51dbucket: bucket-85245key: multipartKey2 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB.java:1104) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.completeMultipartUpload(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.completeMultipartUpload(RpcClient.java:884) s3g_1 | at org.apache.hadoop.ozone.client.OzoneBucket.completeMultipartUpload(OzoneBucket.java:445) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.completeMultipartUpload(ObjectEndpoint.java:498) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | 2019-10-07 08:20:06 ERROR ObjectEndpoint:511 - Error in Complete Multipart Upload Request for bucket: bucket-85245, key: multipartKey3 s3g_1 | MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: volume: s32724f42bcd3225359401cb62da89c51dbucket: bucket-85245key: multipartKey3 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB.java:1104) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) s3g_1 | at com.sun.proxy.$Proxy94.completeMultipartUpload(Unknown Source) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.completeMultipartUpload(RpcClient.java:884) s3g_1 | at org.apache.hadoop.ozone.client.OzoneBucket.completeMultipartUpload(OzoneBucket.java:445) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.completeMultipartUpload(ObjectEndpoint.java:498) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | Oct 07, 2019 8:20:43 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=53, target=172.18.0.8:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47) s3g_1 | at java.base/java.io.InputStream.read(InputStream.java:205) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$0(ObjectEndpoint.java:252) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:20:43 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=49, target=172.18.0.10:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47) s3g_1 | at java.base/java.io.InputStream.read(InputStream.java:205) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$0(ObjectEndpoint.java:252) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:20:43 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=57, target=172.18.0.5:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47) s3g_1 | at java.base/java.io.InputStream.read(InputStream.java:205) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$0(ObjectEndpoint.java:252) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=133, target=172.18.0.8:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2221) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=181, target=172.18.0.8:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47) s3g_1 | at java.base/java.io.InputStream.read(InputStream.java:205) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:563) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:135) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=161, target=172.18.0.10:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.skip(IOUtils.java:2680) s3g_1 | at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2787) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2209) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=177, target=172.18.0.10:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47) s3g_1 | at java.base/java.io.InputStream.read(InputStream.java:205) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123) s3g_1 | at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:563) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:135) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=153, target=172.18.0.10:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.skip(IOUtils.java:2680) s3g_1 | at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2787) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2209) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=141, target=172.18.0.5:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.skip(IOUtils.java:2680) s3g_1 | at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2787) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2209) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=157, target=172.18.0.8:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.skip(IOUtils.java:2680) s3g_1 | at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2787) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2209) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=165, target=172.18.0.10:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2221) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=137, target=172.18.0.5:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.skip(IOUtils.java:2680) s3g_1 | at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2787) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2209) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=145, target=172.18.0.5:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2221) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | Oct 07, 2019 8:21:01 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue s3g_1 | SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=149, target=172.18.0.8:9859} was not shutdown properly!!! ~*~*~* s3g_1 | Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. s3g_1 | java.lang.RuntimeException: ManagedChannel allocation site s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) s3g_1 | at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:181) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:429) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:378) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:291) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:257) s3g_1 | at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:240) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:245) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunk(ChunkInputStream.java:335) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:307) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:259) s3g_1 | at org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:144) s3g_1 | at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:239) s3g_1 | at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171) s3g_1 | at org.apache.hadoop.ozone.s3.io.S3WrapperInputStream.read(S3WrapperInputStream.java:49) s3g_1 | at org.apache.commons.io.IOUtils.skip(IOUtils.java:2680) s3g_1 | at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2787) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2209) s3g_1 | at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2179) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:278) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79) s3g_1 | at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) s3g_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163) s3g_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1135) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:662) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:280) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) s3g_1 | s3g_1 | 2019-10-07 08:21:20 INFO BucketEndpoint:211 - Location is /bucket-28607 s3g_1 | 2019-10-07 08:21:21 INFO BucketEndpoint:211 - Location is /destbucket-83033 s3g_1 | 2019-10-07 08:21:51 INFO BucketEndpoint:211 - Location is /bucket-47644 s3g_1 | 2019-10-07 08:22:30 INFO BucketEndpoint:211 - Location is /bucket-77224 s3g_1 | 2019-10-07 08:22:41 INFO BucketEndpoint:211 - Location is /bucket-26507 s3g_1 | 2019-10-07 08:23:52 INFO BucketEndpoint:211 - Location is /bucket-94819 s3g_1 | 2019-10-07 08:23:57 ERROR OzoneClientProducer:109 - Error: s3g_1 | java.lang.NullPointerException s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:78) s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:71) s3g_1 | at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808) s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61) s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:151) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:93) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:127) s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:70) s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:77) s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:153) s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487) s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:70) s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740) s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710) s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:184) s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:54) s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:129) s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:284) s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:75) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:110) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:113) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:93) s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:62) s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:269) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834)