| Application | \r\n", "Required | \r\n", "Description | \r\n", "
|---|---|---|
| java | \r\n", "Yes | \r\n", "Java 7 or higher is required to run this tool. Please install it from (Java installation) | \r\n", "
| java classpath setup | \r\n", "Yes | \r\n", "If java is not setup in classpath please follow instruction Here to setup. | \r\n", "
\r\n",
"\r\n",
"\r\n",
"Use this tool to test if the Polybase kerberos connection to your kerberos secure HDFS is successfull. This notebook requires about 40 MB of disk space. Learn more about HDFS and polybase kerberos troubleshooting.\r\n",
"## Important Instructions\r\n",
"### **Before you begin, you will need:**\r\n",
"* HDFS cluster name\r\n",
"* Kerberos username\r\n",
"* Kerberos password\r\n",
"* Kerberos realm\r\n",
"* kerberos KDC name or IP address\r\n",
"\r\n",
"This independent tool provide complete details of secure connections which helps administrator to trouble shoot the kerberos connection issue.\r\n",
"(Note: This tool does not require Polybase feature and it does not depend on any SQL server components.)\r\n",
""
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Function definition \r\n",
"### (no user input required)"
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"\r\n",
"def PromptForInfo(promptMsg, errorMsg):\r\n",
" \r\n",
" promptResponse = input(promptMsg)\r\n",
" if promptResponse == \"\":\r\n",
" raise SystemExit(errorMsg + '\\n')\r\n",
" return promptResponse\r\n",
"\r\n",
"def executeCommand(cmd, successMsgs, printMsg):\r\n",
" print(printMsg)\r\n",
" cmdOutput = !{cmd}\r\n",
" cmdOutput = '\\n'.join(cmdOutput)\r\n",
" if any(msg in cmdOutput for msg in successMsgs):\r\n",
" print(f\"\\nKerberos Success \")\r\n",
" else:\r\n",
" if any(msg in cmdOutput for msg in ['java.net.SocketTimeoutException: Receive timed out']):\r\n",
" raise SystemExit(f'\\nFailed due to wrong kdc hostname: Please check the kdc name and rerun the notebook after changing kdc hostname.\\n' + cmd + '\\n output:\\n' + cmdOutput)\r\n",
" if any(msg in cmdOutput for msg in ['Client not found in Kerberos database (6) - CLIENT_NOT_FOUND']):\r\n",
" raise SystemExit(f'\\nThe admin Service Principal supplied does not exist in the realm. Please check if user exist in kerberos realm.\\n' + cmd + '\\n output:\\n' + cmdOutput)\r\n",
" if any(msg in cmdOutput for msg in ['Checksum failed']):\r\n",
" raise SystemExit(f'\\n Bad password. Please check the password and re run the test with right password.\\n' + cmd + '\\n output:\\n' + cmdOutput)\r\n",
" if any(msg in cmdOutput for msg in ['java.io.FileNotFoundException File does not exist']):\r\n",
" raise SystemExit(f'\\n specified file location does not exist on hdfs. change the location.\\n' + cmd + '\\n output:\\n' + cmdOutput)\r\n",
" if any(msg in cmdOutput for msg in ['not recongnized']):\r\n",
" raise SystemExit(f'\\n Java was not installed. Please check pre requisits and rerun this cell.\\n' + cmd + '\\n output:\\n' + cmdOutput)\r\n",
" if any(msg in cmdOutput for msg in ['java.net.ConnectException: Connection refused']):\r\n",
" raise SystemExit(f'\\n Port number is not correct. Please check hdfs connection port number is active.\\n' + cmd + '\\n output:\\n' + cmdOutput)\r\n",
" raise SystemExit(f'\\nFailed during:\\n\\n\\t{cmd}\\n\\nreturned: \\n' + ''.join(cmdOutput) + '.\\n')\r\n",
"\r\n",
"\r\n",
""
],
"metadata": {},
"outputs": [],
"execution_count": 11
},
{
"cell_type": "markdown",
"source": "## Check if Java installed.",
"metadata": {}
},
{
"cell_type": "code",
"source": [
"cmd = f'java -version'\r\n",
"executeCommand(cmd, ['java version', 'build'], 'Checking whether java installed or not.')"
],
"metadata": {},
"outputs": [
{
"name": "stdout",
"text": "Checking whether java installed or not.\n",
"output_type": "stream"
},
{
"name": "stdout",
"text": "\nSuccess >> java -version output:java version \"1.8.0_221\"\nJava(TM) SE Runtime Environment (build 1.8.0_221-b11)\nJava HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)\n",
"output_type": "stream"
}
],
"execution_count": 12
},
{
"cell_type": "markdown",
"source": [
"Enter hdfs hostname: (or IP address)\r\n",
""
],
"metadata": {}
},
{
"cell_type": "code",
"source": [
"# Prompt user inputs:\r\n",
"hdfs_node_name = PromptForInfo('Please provide your cluster hdfs node IP address or host name ', 'Cluster Name node name is required!')"
],
"metadata": {},
"outputs": [],
"execution_count": 13
},
{
"cell_type": "markdown",
"source": "enter HDFS port number: Usually its 8020.",
"metadata": {}
},
{
"cell_type": "code",
"source": "port_number = PromptForInfo('Please provide your Cluster hdfs listening port number', 'port number is required!')",
"metadata": {},
"outputs": [],
"execution_count": 14
},
{
"cell_type": "markdown",
"source": "Enter Kebreros realm name: upper case only (example: EXAMPLE.COM)",
"metadata": {}
},
{
"cell_type": "code",
"source": "kerberos_realm = PromptForInfo('Please provide your Kerberos realm(UPPER CASE) ', 'Kerberos realm is required!')",
"metadata": {},
"outputs": [],
"execution_count": 15
},
{
"cell_type": "markdown",
"source": "Enter KDC server name/IP address (You can find this information in Ambari or hortonworks )",
"metadata": {}
},
{
"cell_type": "code",
"source": "kerberos_kdc = PromptForInfo('Please provide your Kerberos KDC host name or IP ', 'kdc is required!')",
"metadata": {},
"outputs": [],
"execution_count": 16
},
{
"cell_type": "markdown",
"source": "Enter kerberos username: (Example: pdw_user)",
"metadata": {}
},
{
"cell_type": "code",
"source": "kerberos_username = PromptForInfo('Please provide your Kerberos Username for login: ', 'Kerberos Username is required!')",
"metadata": {},
"outputs": [],
"execution_count": 21
},
{
"cell_type": "markdown",
"source": "Enter the file name which contains kerberos password. (You can edit the myKerbPass.txt in current working directory with password and provide myKerbPass.txt as input or enter complete path to the text file where password is stored.)",
"metadata": {}
},
{
"cell_type": "code",
"source": "kerberos_password_file = PromptForInfo('Kerberos Password file name: ', 'Kerberos password file name is required!')",
"metadata": {},
"outputs": [],
"execution_count": 18
},
{
"cell_type": "markdown",
"source": "Run the test",
"metadata": {}
},
{
"cell_type": "code",
"source": [
"\r\n",
"\r\n",
"cmd = f'java -classpath hdfs-kerberos-tester-0.0.1-jar-with-dependencies.jar com.microsoft.kerberos.client.HdfsBridge ' + hdfs_node_name + ' ' + port_number + ' ' + kerberos_realm + ' ' + kerberos_kdc+ ' '+ kerberos_username + ' ' + kerberos_password_file\r\n",
"\r\n",
"executeCommand(cmd, ['CHECKPOINT 2 COMPLETED'], 'Testing Kebreros Connection')\r\n",
""
],
"metadata": {},
"outputs": [
{
"name": "stdout",
"text": "Testing Kebreros Connection\n",
"output_type": "stream"
},
{
"traceback": [
"An exception has occurred, use %tb to see the full traceback.\n",
"\u001b[1;31mSystemExit\u001b[0m\u001b[1;31m:\u001b[0m \nThe admin Service Principal supplied does not exist in the realm. Please check if user exisit in kerberos realm.\njava -classpath hdfs-kerberos-tester-0.0.1-jar-with-dependencies.jar com.microsoft.kerberos.client.HdfsBridge 10.193.27.232 8020 APSHDPMS.COM 10.193.27.236 pdw_user1 myKerbPass.txt\n output:\nCurrent working directory : f:\\azure-data-studio\\notebooks-python\n0 [main] INFO com.microsoft.kerberos.client.HdfsBridge - Hostname: 10.193.27.232\n2 [main] INFO com.microsoft.kerberos.client.HdfsBridge - Port: 8020\n3 [main] INFO com.microsoft.kerberos.client.HdfsBridge - Kerberos Principal: pdw_user1\n3 [main] INFO com.microsoft.kerberos.client.HdfsBridge - Kerberos Password File: myKerbPass.txt\n3 [main] INFO com.microsoft.kerberos.client.HdfsBridge - CHECKPOINT 1 STARTING: Authentication against KDC server\n3 [main] INFO com.microsoft.kerberos.client.HdfsBridge - Filepath to test: /\n805 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS\n806 [main] INFO com.microsoft.kerberos.client.HdfsBridge - Starting to connect to external storage...\nJava config name: null\nNative config name: C:\\windows\\krb5.ini\n>>> KdcAccessibility: reset\nUsing builtin default etypes for default_tkt_enctypes\ndefault etypes for default_tkt_enctypes: 18 17 16 23.\n>>> KrbAsReq creating message\n>>> KrbKdcReq send: kdc=10.193.27.236 UDP:88, timeout=30000, number of retries =3, #bytes=148\n>>> KDCCommunication: kdc=10.193.27.236 UDP:88, timeout=30000,Attempt =1, #bytes=148\n>>> KrbKdcReq send: #bytes read=177\n>>> KdcAccessibility: remove 10.193.27.236\n>>> KDCRep: init() encoding tag is 126 req type is 11\n>>>KRBError:\n\t cTime is Sat Apr 11 16:40:47 PDT 2015 1428795647000\n\t sTime is Thu Sep 05 11:14:55 PDT 2019 1567707295000\n\t suSec is 863876\n\t error code is 6\n\t error Message is Client not found in Kerberos database\n\t cname is pdw_user1@APSHDPMS.COM\n\t sname is krbtgt/APSHDPMS.COM@APSHDPMS.COM\n\t msgType is 30\nKRBError received: CLIENT_NOT_FOUND\njavax.security.auth.login.LoginException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND\n\tat com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)\n\tat com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat javax.security.auth.login.LoginContext.invoke(Unknown Source)\n\tat javax.security.auth.login.LoginContext.access$000(Unknown Source)\n\tat javax.security.auth.login.LoginContext$4.run(Unknown Source)\n\tat javax.security.auth.login.LoginContext$4.run(Unknown Source)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.login.LoginContext.invokePriv(Unknown Source)\n\tat javax.security.auth.login.LoginContext.login(Unknown Source)\n\tat com.microsoft.kerberos.client.KerberosSecureLogin.secureLogin(KerberosSecureLogin.java:89)\n\tat com.microsoft.kerberos.client.HdfsBridge.kerberosLoginHelper(HdfsBridge.java:159)\n\tat com.microsoft.kerberos.client.HdfsBridge.connect(HdfsBridge.java:305)\n\tat com.microsoft.kerberos.client.HdfsBridge.main(HdfsBridge.java:497)\nCaused by: KrbException: Client not found in Kerberos database (6) - CLIENT_NOT_FOUND\n\tat sun.security.krb5.KrbAsRep.