{"cluster": "JIRA-CASSANDRA-102a1b80a28c", "issue_id": "12856696", "key": "CASSANDRA-10102", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: java.lang.UnsupportedOperationException after upgrade to 3.0\n\nUpgrade tests are showing a potential issue. I'm seeing this during rolling upgrades to 3.0 alpha 1, after one node has been upgraded to the alpha.\n\nI will attach cassandra logs here, node1.log is where most of the failures are seen.\n\n{noformat}\nERROR [MessagingService-Incoming-/127.0.0.1] 2015-08-17 12:22:06,888 CassandraDaemon.java:189 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.1,5,main]\njava.lang.UnsupportedOperationException: null\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:485) ~[main/:na]\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:444) ~[main/:na]\n at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:172) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:90) ~[main/:na]\nINFO [GossipStage:1] 2015-08-17 12:22:06,914 StorageService.java:1886 - Node /127.0.0.2 state jump to normal\nERROR [MessagingService-Incoming-/127.0.0.1] 2015-08-17 12:22:06,915 CassandraDaemon.java:189 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.1,5,main]\njava.lang.UnsupportedOperationException: null\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:485) ~[main/:na]\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:444) ~[main/:na]\n at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:172) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:90) ~[main/:na]\n{noformat}\n\nAnother exception showing in logs:\n{noformat}\nERROR [SharedPool-Worker-1] 2015-08-17 12:22:19,358 ErrorMessage.java:336 - Unexpected exception during request\njava.lang.UnsupportedOperationException: Version is 9\n at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serializedSize(PartitionUpdate.java:760) ~[main/:na]\n at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:334) ~[main/:na]\n at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:246) ~[main/:na]\n at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na]\n at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:67) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:587) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:737) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:702) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:1084) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:125) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:942) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:549) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:720) ~[main/:na]\n at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:613) ~[main/:na]\n at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:599) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:204) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:470) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:447) ~[main/:na]\n at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:139) ~[main/:na]\n at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]\n at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]\n at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]\n at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [main/:na]\n at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]\n at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]\n{noformat}\n\nI'm seeing several occurences of both exceptions above.", "query_30": "From: reporter\nSubject: java.lang.UnsupportedOperationException after upgrade to 3.0\n\nUpgrade tests are showing a potential issue. I'm seeing this during rolling upgrades to 3.0 alpha 1, after one node has been upgraded to the alpha.\n\nI will attach cassandra logs here, node1.log is where most of the failures are seen.\n\n{noformat}\nERROR [MessagingService-Incoming-/127.0.0.1] 2015-08-17 12:22:06,888 CassandraDaemon.java:189 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.1,5,main]\njava.lang.UnsupportedOperationException: null\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:485) ~[main/:na]\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:444) ~[main/:na]\n at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:172) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:90) ~[main/:na]\nINFO [GossipStage:1] 2015-08-17 12:22:06,914 StorageService.java:1886 - Node /127.0.0.2 state jump to normal\nERROR [MessagingService-Incoming-/127.0.0.1] 2015-08-17 12:22:06,915 CassandraDaemon.java:189 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.1,5,main]\njava.lang.UnsupportedOperationException: null\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:485) ~[main/:na]\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:444) ~[main/:na]\n at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:172) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:90) ~[main/:na]\n{noformat}\n\nAnother exception showing in logs:\n{noformat}\nERROR [SharedPool-Worker-1] 2015-08-17 12:22:19,358 ErrorMessage.java:336 - Unexpected exception during request\njava.lang.UnsupportedOperationException: Version is 9\n at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serializedSize(PartitionUpdate.java:760) ~[main/:na]\n at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:334) ~[main/:na]\n at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:246) ~[main/:na]\n at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na]\n at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:67) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:587) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:737) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:702) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:1084) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:125) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:942) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:549) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:720) ~[main/:na]\n at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:613) ~[main/:na]\n at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:599) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:204) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:470) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:447) ~[main/:na]\n at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:139) ~[main/:na]\n at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]\n at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]\n at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]\n at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [main/:na]\n at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]\n at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]\n{noformat}\n\nI'm seeing several occurences of both exceptions above.\n\n---\n\nFrom: developer\n\nadding logs", "query_60": "From: reporter\nSubject: java.lang.UnsupportedOperationException after upgrade to 3.0\n\nUpgrade tests are showing a potential issue. I'm seeing this during rolling upgrades to 3.0 alpha 1, after one node has been upgraded to the alpha.\n\nI will attach cassandra logs here, node1.log is where most of the failures are seen.\n\n{noformat}\nERROR [MessagingService-Incoming-/127.0.0.1] 2015-08-17 12:22:06,888 CassandraDaemon.java:189 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.1,5,main]\njava.lang.UnsupportedOperationException: null\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:485) ~[main/:na]\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:444) ~[main/:na]\n at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:172) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:90) ~[main/:na]\nINFO [GossipStage:1] 2015-08-17 12:22:06,914 StorageService.java:1886 - Node /127.0.0.2 state jump to normal\nERROR [MessagingService-Incoming-/127.0.0.1] 2015-08-17 12:22:06,915 CassandraDaemon.java:189 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.1,5,main]\njava.lang.UnsupportedOperationException: null\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:485) ~[main/:na]\n at org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:444) ~[main/:na]\n at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:172) ~[main/:na]\n at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:90) ~[main/:na]\n{noformat}\n\nAnother exception showing in logs:\n{noformat}\nERROR [SharedPool-Worker-1] 2015-08-17 12:22:19,358 ErrorMessage.java:336 - Unexpected exception during request\njava.lang.UnsupportedOperationException: Version is 9\n at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serializedSize(PartitionUpdate.java:760) ~[main/:na]\n at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:334) ~[main/:na]\n at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:246) ~[main/:na]\n at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:166) ~[main/:na]\n at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:67) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:587) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:737) ~[main/:na]\n at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:702) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:1084) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:125) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:942) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:549) ~[main/:na]\n at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:720) ~[main/:na]\n at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:613) ~[main/:na]\n at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:599) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:204) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:470) ~[main/:na]\n at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:447) ~[main/:na]\n at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:139) ~[main/:na]\n at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]\n at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]\n at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]\n at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [main/:na]\n at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]\n at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]\n{noformat}\n\nI'm seeing several occurences of both exceptions above.\n\n---\n\nFrom: developer\n\nadding logs", "target_key": "CASSANDRA-9704"} {"cluster": "JIRA-CASSANDRA-18d3ddb47d49", "issue_id": "12958619", "key": "CASSANDRA-11570", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Concurrent execution of prepared statement returns invalid JSON as result\n\nWhen I use prepared statement for async execution of multiple statements I get JSON with broken data. Keys got totally corrupted when values seems to be normal though.\n\nFirst I encoutered this issue when I were performing stress testing of our project using custom script. We are using DataStax C++ driver and execute statements from different fibers.\n\nThen I was trying to isolate problem and wrote simple C# program which starts multiple Tasks in a loop. Each task uses the once created prepared statement to read data from the base. As you can see results are totally mess.\n\nI 've attached archive with console C# project (1 cs file) which just print resulting JSON to user. \nHere is the main part of C# code.\n\n{noformat}\nstatic void Main(string[] args)\n{\n const int task_count = 300;\n\n using(var cluster = Cluster.Builder().AddContactPoints(/*contact points here*/).Build())\n {\n using(var session = cluster.Connect())\n {\n var prepared = session.Prepare(\"select json * from test_neptunao.ubuntu where id=?\");\n var tasks = new Task[task_count];\n for(int i = 0; i < task_count; i++)\n {\n tasks[i] = Query(prepared, session);\n }\n Task.WaitAll(tasks);\n }\n }\n Console.ReadKey();\n}\n\nprivate static Task Query(PreparedStatement prepared, ISession session)\n{\n string id = GetIdOfRandomRow();\n var stmt = prepared.Bind(id);\n stmt.SetConsistencyLevel(ConsistencyLevel.One);\n return session.ExecuteAsync(stmt).ContinueWith(tr =>\n {\n foreach(var row in tr.Result)\n {\n var value = row.GetValue(0);\n //some kind of output\n }\n });\n}\n{noformat}\n\nI also attached cql script with test DB schema.\n\n{noformat}\nCREATE KEYSPACE IF NOT EXISTS test_neptunao\nWITH replication = {\n\t'class' : 'SimpleStrategy',\n\t'replication_factor' : 3\n};\n\nuse test_neptunao;\n\ncreate table if not exists ubuntu (\n\tid timeuuid PRIMARY KEY,\n\tprecise_pangolin text,\n\ttrusty_tahr text,\n\twily_werewolf text,\t\n\tvivid_vervet text,\n\tsaucy_salamander text,\n\tlucid_lynx text\n);\n{noformat}", "query_30": "From: reporter\nSubject: Concurrent execution of prepared statement returns invalid JSON as result\n\nWhen I use prepared statement for async execution of multiple statements I get JSON with broken data. Keys got totally corrupted when values seems to be normal though.\n\nFirst I encoutered this issue when I were performing stress testing of our project using custom script. We are using DataStax C++ driver and execute statements from different fibers.\n\nThen I was trying to isolate problem and wrote simple C# program which starts multiple Tasks in a loop. Each task uses the once created prepared statement to read data from the base. As you can see results are totally mess.\n\nI 've attached archive with console C# project (1 cs file) which just print resulting JSON to user. \nHere is the main part of C# code.\n\n{noformat}\nstatic void Main(string[] args)\n{\n const int task_count = 300;\n\n using(var cluster = Cluster.Builder().AddContactPoints(/*contact points here*/).Build())\n {\n using(var session = cluster.Connect())\n {\n var prepared = session.Prepare(\"select json * from test_neptunao.ubuntu where id=?\");\n var tasks = new Task[task_count];\n for(int i = 0; i < task_count; i++)\n {\n tasks[i] = Query(prepared, session);\n }\n Task.WaitAll(tasks);\n }\n }\n Console.ReadKey();\n}\n\nprivate static Task Query(PreparedStatement prepared, ISession session)\n{\n string id = GetIdOfRandomRow();\n var stmt = prepared.Bind(id);\n stmt.SetConsistencyLevel(ConsistencyLevel.One);\n return session.ExecuteAsync(stmt).ContinueWith(tr =>\n {\n foreach(var row in tr.Result)\n {\n var value = row.GetValue(0);\n //some kind of output\n }\n });\n}\n{noformat}\n\nI also attached cql script with test DB schema.\n\n{noformat}\nCREATE KEYSPACE IF NOT EXISTS test_neptunao\nWITH replication = {\n\t'class' : 'SimpleStrategy',\n\t'replication_factor' : 3\n};\n\nuse test_neptunao;\n\ncreate table if not exists ubuntu (\n\tid timeuuid PRIMARY KEY,\n\tprecise_pangolin text,\n\ttrusty_tahr text,\n\twily_werewolf text,\t\n\tvivid_vervet text,\n\tsaucy_salamander text,\n\tlucid_lynx text\n);\n{noformat}\n\n---\n\nFrom: developer\n\nI've changed the samle a bit to fetch only one row per query. \n\nI've also attached the broken and expected outputs. You can see invalid result in the row with id `516b00a2-01a7-11e6-8630-c04f49e62c6b` for example.", "query_60": "From: reporter\nSubject: Concurrent execution of prepared statement returns invalid JSON as result\n\nWhen I use prepared statement for async execution of multiple statements I get JSON with broken data. Keys got totally corrupted when values seems to be normal though.\n\nFirst I encoutered this issue when I were performing stress testing of our project using custom script. We are using DataStax C++ driver and execute statements from different fibers.\n\nThen I was trying to isolate problem and wrote simple C# program which starts multiple Tasks in a loop. Each task uses the once created prepared statement to read data from the base. As you can see results are totally mess.\n\nI 've attached archive with console C# project (1 cs file) which just print resulting JSON to user. \nHere is the main part of C# code.\n\n{noformat}\nstatic void Main(string[] args)\n{\n const int task_count = 300;\n\n using(var cluster = Cluster.Builder().AddContactPoints(/*contact points here*/).Build())\n {\n using(var session = cluster.Connect())\n {\n var prepared = session.Prepare(\"select json * from test_neptunao.ubuntu where id=?\");\n var tasks = new Task[task_count];\n for(int i = 0; i < task_count; i++)\n {\n tasks[i] = Query(prepared, session);\n }\n Task.WaitAll(tasks);\n }\n }\n Console.ReadKey();\n}\n\nprivate static Task Query(PreparedStatement prepared, ISession session)\n{\n string id = GetIdOfRandomRow();\n var stmt = prepared.Bind(id);\n stmt.SetConsistencyLevel(ConsistencyLevel.One);\n return session.ExecuteAsync(stmt).ContinueWith(tr =>\n {\n foreach(var row in tr.Result)\n {\n var value = row.GetValue(0);\n //some kind of output\n }\n });\n}\n{noformat}\n\nI also attached cql script with test DB schema.\n\n{noformat}\nCREATE KEYSPACE IF NOT EXISTS test_neptunao\nWITH replication = {\n\t'class' : 'SimpleStrategy',\n\t'replication_factor' : 3\n};\n\nuse test_neptunao;\n\ncreate table if not exists ubuntu (\n\tid timeuuid PRIMARY KEY,\n\tprecise_pangolin text,\n\ttrusty_tahr text,\n\twily_werewolf text,\t\n\tvivid_vervet text,\n\tsaucy_salamander text,\n\tlucid_lynx text\n);\n{noformat}\n\n---\n\nFrom: developer\n\nI've changed the samle a bit to fetch only one row per query. \n\nI've also attached the broken and expected outputs. You can see invalid result in the row with id `516b00a2-01a7-11e6-8630-c04f49e62c6b` for example.", "target_key": "CASSANDRA-11048"} {"cluster": "JIRA-CASSANDRA-88571d61eb49", "issue_id": "12978647", "key": "CASSANDRA-12003", "n_comments": 6, "progress_valid": {"0": true, "30": true, "60": true}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Not filtering Data completely when where clause is applied to clustering column\n\nOn issuing this query:;\n\nselect id,col1 from table where id=2134 and col1='clust_key1';\n\n id | col1\n------+----------------------\n 2429 | clust_key2\n 2429 | clust_key3\n------------more rows-------\n 2429 | clust_key1\n------------more rows-------\n 2429 | clust_key4\n 2429 | clust_key5\n\n(16 rows)\n\nWhereas only one result was expected (Row bearing col1 - clust_key1), we got that result but along with 15 other unexpected rows..\n\nTotal number of rows in the partition are 20 (Verified using select id,col1 from table where id=2134; ) as well as json dump. We are wondering why Cassandra could not filter the results completely. I have checked that the data is intact by taking json dump and validating using sstabledump tool. \n", "query_30": "From: reporter\nSubject: Not filtering Data completely when where clause is applied to clustering column\n\nOn issuing this query:;\n\nselect id,col1 from table where id=2134 and col1='clust_key1';\n\n id | col1\n------+----------------------\n 2429 | clust_key2\n 2429 | clust_key3\n------------more rows-------\n 2429 | clust_key1\n------------more rows-------\n 2429 | clust_key4\n 2429 | clust_key5\n\n(16 rows)\n\nWhereas only one result was expected (Row bearing col1 - clust_key1), we got that result but along with 15 other unexpected rows..\n\nTotal number of rows in the partition are 20 (Verified using select id,col1 from table where id=2134; ) as well as json dump. We are wondering why Cassandra could not filter the results completely. I have checked that the data is intact by taking json dump and validating using sstabledump tool. \n\n\n---\n\nFrom: developer\n\nIs it possible to attach the full schema of this table? Details like column names shouldn't be important, so feel free to anonymize those as appropriate.\n\n---\n\nFrom: developer\n\nHi Joel,\n\nI have attached the schema for the table. \nLet me know if any other input is required. I am debugging at our end as well, will update if anything substantial comes out.", "query_60": "From: reporter\nSubject: Not filtering Data completely when where clause is applied to clustering column\n\nOn issuing this query:;\n\nselect id,col1 from table where id=2134 and col1='clust_key1';\n\n id | col1\n------+----------------------\n 2429 | clust_key2\n 2429 | clust_key3\n------------more rows-------\n 2429 | clust_key1\n------------more rows-------\n 2429 | clust_key4\n 2429 | clust_key5\n\n(16 rows)\n\nWhereas only one result was expected (Row bearing col1 - clust_key1), we got that result but along with 15 other unexpected rows..\n\nTotal number of rows in the partition are 20 (Verified using select id,col1 from table where id=2134; ) as well as json dump. We are wondering why Cassandra could not filter the results completely. I have checked that the data is intact by taking json dump and validating using sstabledump tool. \n\n\n---\n\nFrom: developer\n\nIs it possible to attach the full schema of this table? Details like column names shouldn't be important, so feel free to anonymize those as appropriate.\n\n---\n\nFrom: developer\n\nHi Joel,\n\nI have attached the schema for the table. \nLet me know if any other input is required. I am debugging at our end as well, will update if anything substantial comes out.\n\n---\n\nFrom: developer\n\nI've got no great ideas here - either a repro or the affected sstables is probably what someone needs to debug this further.\n\n---\n\nFrom: developer\n\nIn the mailing list it looked like you might be using SASI with LIKE-style queries ({{WHERE id=2134 AND col1 = '\\*value\\*'}}), is that the case? If so it looks like your schema and example case might be incomplete here, so more details would be helpful.", "target_key": "CASSANDRA-11513"} {"cluster": "JIRA-CASSANDRA-c72ee123687a", "issue_id": "13004584", "key": "CASSANDRA-12633", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Assert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.\n\nAssert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.\n\nThere is a possibility of assert errors in ByteBufferUtil.writeWithShortLength which can cause a stop to processing. There is already good error handling it just needs to catch Throwable instead of Exception at https://github.com/apache/cassandra/blob/c26bd91852cbf19d7dba9f62078f5da31b04dbe0/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L239 \n\nAn example of when this happens:\n\n{code}\nERROR [MessagingService-Outgoing-/10.0.19.152] 2016-05-03 21:43:54,638 CassandraDaemon.java:227 - Exception in thread Thread[MessagingService-Outgoing-/10.0.19.152,5,main]\njava.lang.AssertionError: 103265\n at org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:392) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:381) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:271) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:259) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:503) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:490) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromReadCommand.java:168) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:330) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:282) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.1.12.jar:2.1.12]\n{code}\n\nThe exact error happened when you sent a read request via cql for a clustering column value with more than 64k chars much like CASSANDRA-8280 but on reads. That exact error case is no longer reproducable in the latest versions of C* but in general it seems like it should still handle the possible asserts.", "query_30": "From: reporter\nSubject: Assert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.\n\nAssert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.\n\nThere is a possibility of assert errors in ByteBufferUtil.writeWithShortLength which can cause a stop to processing. There is already good error handling it just needs to catch Throwable instead of Exception at https://github.com/apache/cassandra/blob/c26bd91852cbf19d7dba9f62078f5da31b04dbe0/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L239 \n\nAn example of when this happens:\n\n{code}\nERROR [MessagingService-Outgoing-/10.0.19.152] 2016-05-03 21:43:54,638 CassandraDaemon.java:227 - Exception in thread Thread[MessagingService-Outgoing-/10.0.19.152,5,main]\njava.lang.AssertionError: 103265\n at org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:392) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:381) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:271) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:259) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:503) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:490) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromReadCommand.java:168) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:330) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:282) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.1.12.jar:2.1.12]\n{code}\n\nThe exact error happened when you sent a read request via cql for a clustering column value with more than 64k chars much like CASSANDRA-8280 but on reads. That exact error case is no longer reproducable in the latest versions of C* but in general it seems like it should still handle the possible asserts.\n\n---\n\nFrom: developer\n\nStarted at 2.1 but if it isn't critical enough for that I made a patch with it starting at 2.2 as well. Let me know which is preferred and I can create patches for the other branches.", "query_60": "From: reporter\nSubject: Assert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.\n\nAssert Errors occurring with OutboundTcpConnection causes Cassandra to slowly stop functioning.\n\nThere is a possibility of assert errors in ByteBufferUtil.writeWithShortLength which can cause a stop to processing. There is already good error handling it just needs to catch Throwable instead of Exception at https://github.com/apache/cassandra/blob/c26bd91852cbf19d7dba9f62078f5da31b04dbe0/src/java/org/apache/cassandra/net/OutboundTcpConnection.java#L239 \n\nAn example of when this happens:\n\n{code}\nERROR [MessagingService-Outgoing-/10.0.19.152] 2016-05-03 21:43:54,638 CassandraDaemon.java:227 - Exception in thread Thread[MessagingService-Outgoing-/10.0.19.152,5,main]\njava.lang.AssertionError: 103265\n at org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:392) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:381) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:271) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:259) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:503) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:490) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromReadCommand.java:168) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:330) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:282) ~[apache-cassandra-2.1.12.jar:2.1.12]\n at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218) ~[apache-cassandra-2.1.12.jar:2.1.12]\n{code}\n\nThe exact error happened when you sent a read request via cql for a clustering column value with more than 64k chars much like CASSANDRA-8280 but on reads. That exact error case is no longer reproducable in the latest versions of C* but in general it seems like it should still handle the possible asserts.\n\n---\n\nFrom: developer\n\nStarted at 2.1 but if it isn't critical enough for that I made a patch with it starting at 2.2 as well. Let me know which is preferred and I can create patches for the other branches.\n\n---\n\nFrom: developer\n\nPushing up to my github for CI, let's make sure tests are happy.\n\n|[trunk|https://github.com/jeffjirsa/cassandra/tree/cassandra-12633]|[testall|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-testall/]|[dtest|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-dtest/]|\n|[3.X|https://github.com/jeffjirsa/cassandra/tree/cassandra-12633-3.X]|[testall|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-testall-3.X/]|[dtest|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-dtest-3.X/]|\n|[3.0|https://github.com/jeffjirsa/cassandra/tree/cassandra-12633-3.0]|[testall|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-testall-3.0/]|[dtest|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-dtest-3.0/]|\n|[2.2|https://github.com/jeffjirsa/cassandra/tree/cassandra-12633-2.2]|[testall|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-testall-2.2/]|[dtest|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-dtest-2.2/]|\n|[2.1|https://github.com/jeffjirsa/cassandra/tree/cassandra-12633-2.1]|[testall|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-testall-2.1/]|[dtest|http://cassci.datastax.com/job/jeffjirsa-cassandra-12633-dtest-2.1/]|\n", "target_key": "CASSANDRA-11882"} {"cluster": "JIRA-CASSANDRA-d001181e0a40", "issue_id": "13134601", "key": "CASSANDRA-14199", "n_comments": 3, "progress_valid": {"0": true, "30": true, "60": true}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: exception when dumping sstable with frozen collection of UUID\n\nWhen dumping (sstabledump) sstable with frozen collection of UUID, there is exception like this:\r\n{code:java}\r\nException in thread \"main\" org.apache.cassandra.serializers.MarshalException: UUID should be 16 or 0 bytes (24)\r\n at org.apache.cassandra.serializers.UUIDSerializer.validate(UUIDSerializer.java:43)\r\n at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:128)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeCell(JsonTransformer.java:440)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeColumnData(JsonTransformer.java:374)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeRow(JsonTransformer.java:278)\r\n at org.apache.cassandra.tools.JsonTransformer.serializePartition(JsonTransformer.java:213)\r\n at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)\r\n at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\r\n at java.util.Iterator.forEachRemaining(Iterator.java:116)\r\n at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)\r\n at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\r\n at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\r\n at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)\r\n at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)\r\n at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\r\n at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)\r\n at org.apache.cassandra.tools.JsonTransformer.toJson(JsonTransformer.java:102)\r\n at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:242){code}\r\n \r\n\r\n*Steps to reproduce:*\r\n{code:java}\r\ncqlsh> create TABLE stresscql.sstabledump_test(userid text PRIMARY KEY, c1 list, c2 frozen>, c3 set, c4 frozen>, c5 map, c6 frozen>);\r\ncqlsh> insert INTO stresscql.sstabledump_test (userid, c1, c2, c3, c4, c5, c6) VALUES ( 'id', [6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4], [6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4], {'set', 'user'}, {'view', 'over'}, {'good': 'hello', 'root': 'text'}, {'driver': 'java', 'note': 'new'});{code}\r\n \r\n\r\n*Root cause:*\r\n\r\nFrozen collection is treated as simple column and it's the client's responsibility to parse the data from ByteBuffer. We have this logic in different drivers but sstabledump doesn't have the logic in place. It just treat the whole collection as a single UUID.", "query_30": "From: reporter\nSubject: exception when dumping sstable with frozen collection of UUID\n\nWhen dumping (sstabledump) sstable with frozen collection of UUID, there is exception like this:\r\n{code:java}\r\nException in thread \"main\" org.apache.cassandra.serializers.MarshalException: UUID should be 16 or 0 bytes (24)\r\n at org.apache.cassandra.serializers.UUIDSerializer.validate(UUIDSerializer.java:43)\r\n at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:128)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeCell(JsonTransformer.java:440)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeColumnData(JsonTransformer.java:374)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeRow(JsonTransformer.java:278)\r\n at org.apache.cassandra.tools.JsonTransformer.serializePartition(JsonTransformer.java:213)\r\n at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)\r\n at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\r\n at java.util.Iterator.forEachRemaining(Iterator.java:116)\r\n at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)\r\n at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\r\n at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\r\n at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)\r\n at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)\r\n at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\r\n at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)\r\n at org.apache.cassandra.tools.JsonTransformer.toJson(JsonTransformer.java:102)\r\n at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:242){code}\r\n \r\n\r\n*Steps to reproduce:*\r\n{code:java}\r\ncqlsh> create TABLE stresscql.sstabledump_test(userid text PRIMARY KEY, c1 list, c2 frozen>, c3 set, c4 frozen>, c5 map, c6 frozen>);\r\ncqlsh> insert INTO stresscql.sstabledump_test (userid, c1, c2, c3, c4, c5, c6) VALUES ( 'id', [6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4], [6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4], {'set', 'user'}, {'view', 'over'}, {'good': 'hello', 'root': 'text'}, {'driver': 'java', 'note': 'new'});{code}\r\n \r\n\r\n*Root cause:*\r\n\r\nFrozen collection is treated as simple column and it's the client's responsibility to parse the data from ByteBuffer. We have this logic in different drivers but sstabledump doesn't have the logic in place. It just treat the whole collection as a single UUID.\n\n---\n\nFrom: developer\n\nI pushed a fix for 3.0 [here|https://github.com/szhou1234/cassandra/commit/1a8fff02e93e0acb90e785fdca7f30d9aae54b1a] and will provide fixes for newer versions.\r\n\r\n[~cnlwsu] could you help review this? For your convenience, this is how the output looks like after the fix:\r\n{code}\r\n[\r\n {\r\n \"partition\" : {\r\n \"key\" : [ \"id\" ],\r\n \"position\" : 0\r\n },\r\n \"rows\" : [\r\n {\r\n \"type\" : \"row\",\r\n \"position\" : 181,\r\n \"liveness_info\" : { \"tstamp\" : \"2018-01-29T22:58:49.111820Z\" },\r\n \"cells\" : [\r\n { \"name\" : \"c2\", \"frozen\" : true, \"values\" : [ \"3\", \"4\" ] },\r\n { \"name\" : \"c4\", \"frozen\" : true, \"values\" : [ \"over\", \"view\" ] },\r\n { \"name\" : \"c6\", \"frozen\" : true, \"values\" : { \"driver\" : \"java\", \"note\" : \"new\" } },\r\n { \"name\" : \"c1\", \"deletion_info\" : { \"marked_deleted\" : \"2018-01-29T22:58:49.111819Z\", \"local_delete_time\" : \"2018-01-29T22:58:49Z\" } },\r\n { \"name\" : \"c1\", \"path\" : [ \"f7b01890-0547-11e8-817b-adb40ecebcf5\" ] },\r\n { \"name\" : \"c1\", \"path\" : [ \"f7b01891-0547-11e8-817b-adb40ecebcf5\" ] },\r\n { \"name\" : \"c3\", \"deletion_info\" : { \"marked_deleted\" : \"2018-01-29T22:58:49.111819Z\", \"local_delete_time\" : \"2018-01-29T22:58:49Z\" } },\r\n { \"name\" : \"c3\", \"path\" : [ \"set\" ] },\r\n { \"name\" : \"c3\", \"path\" : [ \"user\" ] },\r\n { \"name\" : \"c5\", \"deletion_info\" : { \"marked_deleted\" : \"2018-01-29T22:58:49.111819Z\", \"local_delete_time\" : \"2018-01-29T22:58:49Z\" } },\r\n { \"name\" : \"c5\", \"path\" : [ \"good\" ] },\r\n { \"name\" : \"c5\", \"path\" : [ \"root\" ] }\r\n ]\r\n }\r\n ]\r\n }\r\n]\r\n{code}\r\n\r\nTwo changes:\r\n- I added field \"frozen\" for frozen collections.\r\n- The elements in the frozen collection will be in one line (other than one line per each element in un-frozen collection), to better indicate that they are immutable.\r\n\r\nThere could be another independent issue that, for un-frozen collection, there is always one output line for \"deletion_info\", even the cell doesn't have any deletion. Anyway there should be a separate fix if it's an issue.", "query_60": "From: reporter\nSubject: exception when dumping sstable with frozen collection of UUID\n\nWhen dumping (sstabledump) sstable with frozen collection of UUID, there is exception like this:\r\n{code:java}\r\nException in thread \"main\" org.apache.cassandra.serializers.MarshalException: UUID should be 16 or 0 bytes (24)\r\n at org.apache.cassandra.serializers.UUIDSerializer.validate(UUIDSerializer.java:43)\r\n at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:128)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeCell(JsonTransformer.java:440)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeColumnData(JsonTransformer.java:374)\r\n at org.apache.cassandra.tools.JsonTransformer.serializeRow(JsonTransformer.java:278)\r\n at org.apache.cassandra.tools.JsonTransformer.serializePartition(JsonTransformer.java:213)\r\n at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)\r\n at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)\r\n at java.util.Iterator.forEachRemaining(Iterator.java:116)\r\n at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)\r\n at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\r\n at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\r\n at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)\r\n at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)\r\n at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\r\n at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)\r\n at org.apache.cassandra.tools.JsonTransformer.toJson(JsonTransformer.java:102)\r\n at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:242){code}\r\n \r\n\r\n*Steps to reproduce:*\r\n{code:java}\r\ncqlsh> create TABLE stresscql.sstabledump_test(userid text PRIMARY KEY, c1 list, c2 frozen>, c3 set, c4 frozen>, c5 map, c6 frozen>);\r\ncqlsh> insert INTO stresscql.sstabledump_test (userid, c1, c2, c3, c4, c5, c6) VALUES ( 'id', [6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4], [6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4], {'set', 'user'}, {'view', 'over'}, {'good': 'hello', 'root': 'text'}, {'driver': 'java', 'note': 'new'});{code}\r\n \r\n\r\n*Root cause:*\r\n\r\nFrozen collection is treated as simple column and it's the client's responsibility to parse the data from ByteBuffer. We have this logic in different drivers but sstabledump doesn't have the logic in place. It just treat the whole collection as a single UUID.\n\n---\n\nFrom: developer\n\nI pushed a fix for 3.0 [here|https://github.com/szhou1234/cassandra/commit/1a8fff02e93e0acb90e785fdca7f30d9aae54b1a] and will provide fixes for newer versions.\r\n\r\n[~cnlwsu] could you help review this? For your convenience, this is how the output looks like after the fix:\r\n{code}\r\n[\r\n {\r\n \"partition\" : {\r\n \"key\" : [ \"id\" ],\r\n \"position\" : 0\r\n },\r\n \"rows\" : [\r\n {\r\n \"type\" : \"row\",\r\n \"position\" : 181,\r\n \"liveness_info\" : { \"tstamp\" : \"2018-01-29T22:58:49.111820Z\" },\r\n \"cells\" : [\r\n { \"name\" : \"c2\", \"frozen\" : true, \"values\" : [ \"3\", \"4\" ] },\r\n { \"name\" : \"c4\", \"frozen\" : true, \"values\" : [ \"over\", \"view\" ] },\r\n { \"name\" : \"c6\", \"frozen\" : true, \"values\" : { \"driver\" : \"java\", \"note\" : \"new\" } },\r\n { \"name\" : \"c1\", \"deletion_info\" : { \"marked_deleted\" : \"2018-01-29T22:58:49.111819Z\", \"local_delete_time\" : \"2018-01-29T22:58:49Z\" } },\r\n { \"name\" : \"c1\", \"path\" : [ \"f7b01890-0547-11e8-817b-adb40ecebcf5\" ] },\r\n { \"name\" : \"c1\", \"path\" : [ \"f7b01891-0547-11e8-817b-adb40ecebcf5\" ] },\r\n { \"name\" : \"c3\", \"deletion_info\" : { \"marked_deleted\" : \"2018-01-29T22:58:49.111819Z\", \"local_delete_time\" : \"2018-01-29T22:58:49Z\" } },\r\n { \"name\" : \"c3\", \"path\" : [ \"set\" ] },\r\n { \"name\" : \"c3\", \"path\" : [ \"user\" ] },\r\n { \"name\" : \"c5\", \"deletion_info\" : { \"marked_deleted\" : \"2018-01-29T22:58:49.111819Z\", \"local_delete_time\" : \"2018-01-29T22:58:49Z\" } },\r\n { \"name\" : \"c5\", \"path\" : [ \"good\" ] },\r\n { \"name\" : \"c5\", \"path\" : [ \"root\" ] }\r\n ]\r\n }\r\n ]\r\n }\r\n]\r\n{code}\r\n\r\nTwo changes:\r\n- I added field \"frozen\" for frozen collections.\r\n- The elements in the frozen collection will be in one line (other than one line per each element in un-frozen collection), to better indicate that they are immutable.\r\n\r\nThere could be another independent issue that, for un-frozen collection, there is always one output line for \"deletion_info\", even the cell doesn't have any deletion. Anyway there should be a separate fix if it's an issue.\n\n---\n\nFrom: developer\n\nthis is fixed on trunk it looks like\r\n\r\n{code}\r\n[\r\n {\r\n \"partition\" : {\r\n \"key\" : [ \"id\" ],\r\n \"position\" : 0\r\n },\r\n \"rows\" : [\r\n {\r\n \"type\" : \"row\",\r\n \"position\" : 186,\r\n \"liveness_info\" : { \"tstamp\" : \"2018-02-14T03:57:41.791250Z\" },\r\n \"cells\" : [\r\n { \"name\" : \"c2\", \"value\" : [\"6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4\"] },\r\n { \"name\" : \"c4\", \"value\" : [\"over\", \"view\"] },\r\n { \"name\" : \"c6\", \"value\" : {\"driver\": \"java\", \"note\": \"new\"} },\r\n { \"name\" : \"c1\", \"deletion_info\" : { \"marked_deleted\" : \"2018-02-14T03:57:41.791249Z\", \"local_delete_time\" : \"2018-02-14T03:57:41Z\" } },\r\n { \"name\" : \"c1\", \"path\" : [ \"34991370-113b-11e8-9d5e-515f8b1c000b\" ], \"value\" : \"6947e8c0-02fa-11e8-87e1-fb0d0e20b5c4\" },\r\n { \"name\" : \"c3\", \"deletion_info\" : { \"marked_deleted\" : \"2018-02-14T03:57:41.791249Z\", \"local_delete_time\" : \"2018-02-14T03:57:41Z\" } },\r\n { \"name\" : \"c3\", \"path\" : [ \"set\" ], \"value\" : \"\" },\r\n { \"name\" : \"c3\", \"path\" : [ \"user\" ], \"value\" : \"\" },\r\n { \"name\" : \"c5\", \"deletion_info\" : { \"marked_deleted\" : \"2018-02-14T03:57:41.791249Z\", \"local_delete_time\" : \"2018-02-14T03:57:41Z\" } },\r\n { \"name\" : \"c5\", \"path\" : [ \"good\" ], \"value\" : \"hello\" },\r\n { \"name\" : \"c5\", \"path\" : [ \"root\" ], \"value\" : \"text\" }\r\n ]\r\n }\r\n ]\r\n }\r\n]\r\n{code}", "target_key": "CASSANDRA-13573"} {"cluster": "JIRA-CASSANDRA-c6b13e0fa922", "issue_id": "13158783", "key": "CASSANDRA-14444", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Got NPE when querying Cassandra 3.11.2\n\nWe just upgraded our Cassandra cluster from 2.2.6 to 3.11.2\r\n\r\nAfter upgrading, we immediately got exceptions in Cassandra like this one: \r\n\r\n \r\n{code}\r\nERROR [Native-Transport-Requests-1] 2018-05-11 17:10:21,994 QueryMessage.java:129 - Unexpected error during query\r\njava.lang.NullPointerException: null\r\nat org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:248) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:92) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.service.pager.PartitionRangeQueryPager.(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:268) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:475) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:288) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:118) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]\r\nat org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]\r\n{code}\r\n \r\n\r\nThe table schema is like:\r\n{code}\r\nCREATE TABLE example.example_table (\r\n id bigint,\r\n hash text,\r\n json text,\r\n PRIMARY KEY (id, hash)\r\n) WITH COMPACT STORAGE\r\n{code}\r\n \r\n\r\nThe query is something like:\r\n{code}\r\n\"select * from example.example_table;\" // (We do know this is bad practise, and we are trying to fix that right now)\r\n{code}\r\nwith fetch-size as 200, using DataStax Java driver. \r\n\r\nThis table contains about 20k rows. \r\n\r\n \r\n\r\nActually, the fix is quite simple, \r\n\r\n \r\n{code}\r\n--- a/src/java/org/apache/cassandra/service/pager/PagingState.java\r\n+++ b/src/java/org/apache/cassandra/service/pager/PagingState.java\r\n@@ -46,7 +46,7 @@ public class PagingState\r\n\r\npublic PagingState(ByteBuffer partitionKey, RowMark rowMark, int remaining, int remainingInPartition)\r\n {\r\n- this.partitionKey = partitionKey;\r\n+ this.partitionKey = partitionKey == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : partitionKey;\r\n this.rowMark = rowMark;\r\n this.remaining = remaining;\r\n this.remainingInPartition = remainingInPartition;\r\n{code}\r\n \r\n\r\n\"partitionKey == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : partitionKey;\" is in 2.2.6 and 2.2.8. But it was removed for some reason. \r\n\r\nThe interesting part is that, we have: \r\n{code}\r\npublic final ByteBuffer partitionKey; // Can be null for single partition queries.\r\n{code}\r\nIt seems \"partitionKey\" could be null.\r\n\r\nThanks a lot. \r\n\r\n \r\n\r\n \r\n\r\n ", "query_30": "From: reporter\nSubject: Got NPE when querying Cassandra 3.11.2\n\nWe just upgraded our Cassandra cluster from 2.2.6 to 3.11.2\r\n\r\nAfter upgrading, we immediately got exceptions in Cassandra like this one: \r\n\r\n \r\n{code}\r\nERROR [Native-Transport-Requests-1] 2018-05-11 17:10:21,994 QueryMessage.java:129 - Unexpected error during query\r\njava.lang.NullPointerException: null\r\nat org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:248) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:92) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.service.pager.PartitionRangeQueryPager.(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:268) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:475) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:288) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:118) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]\r\nat org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]\r\n{code}\r\n \r\n\r\nThe table schema is like:\r\n{code}\r\nCREATE TABLE example.example_table (\r\n id bigint,\r\n hash text,\r\n json text,\r\n PRIMARY KEY (id, hash)\r\n) WITH COMPACT STORAGE\r\n{code}\r\n \r\n\r\nThe query is something like:\r\n{code}\r\n\"select * from example.example_table;\" // (We do know this is bad practise, and we are trying to fix that right now)\r\n{code}\r\nwith fetch-size as 200, using DataStax Java driver. \r\n\r\nThis table contains about 20k rows. \r\n\r\n \r\n\r\nActually, the fix is quite simple, \r\n\r\n \r\n{code}\r\n--- a/src/java/org/apache/cassandra/service/pager/PagingState.java\r\n+++ b/src/java/org/apache/cassandra/service/pager/PagingState.java\r\n@@ -46,7 +46,7 @@ public class PagingState\r\n\r\npublic PagingState(ByteBuffer partitionKey, RowMark rowMark, int remaining, int remainingInPartition)\r\n {\r\n- this.partitionKey = partitionKey;\r\n+ this.partitionKey = partitionKey == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : partitionKey;\r\n this.rowMark = rowMark;\r\n this.remaining = remaining;\r\n this.remainingInPartition = remainingInPartition;\r\n{code}\r\n \r\n\r\n\"partitionKey == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : partitionKey;\" is in 2.2.6 and 2.2.8. But it was removed for some reason. \r\n\r\nThe interesting part is that, we have: \r\n{code}\r\npublic final ByteBuffer partitionKey; // Can be null for single partition queries.\r\n{code}\r\nIt seems \"partitionKey\" could be null.\r\n\r\nThanks a lot. \r\n\r\n \r\n\r\n \r\n\r\n \n\n---\n\nFrom: developer\n\nHere is my PR: [https://github.com/apache/cassandra/pull/225]\r\n\r\nCould anyone please have a look? Thanks a lot. ", "query_60": "From: reporter\nSubject: Got NPE when querying Cassandra 3.11.2\n\nWe just upgraded our Cassandra cluster from 2.2.6 to 3.11.2\r\n\r\nAfter upgrading, we immediately got exceptions in Cassandra like this one: \r\n\r\n \r\n{code}\r\nERROR [Native-Transport-Requests-1] 2018-05-11 17:10:21,994 QueryMessage.java:129 - Unexpected error during query\r\njava.lang.NullPointerException: null\r\nat org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:248) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:92) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.service.pager.PartitionRangeQueryPager.(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:268) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:475) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:288) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:118) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]\r\nat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]\r\nat org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.11.2.jar:3.11.2]\r\nat java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]\r\n{code}\r\n \r\n\r\nThe table schema is like:\r\n{code}\r\nCREATE TABLE example.example_table (\r\n id bigint,\r\n hash text,\r\n json text,\r\n PRIMARY KEY (id, hash)\r\n) WITH COMPACT STORAGE\r\n{code}\r\n \r\n\r\nThe query is something like:\r\n{code}\r\n\"select * from example.example_table;\" // (We do know this is bad practise, and we are trying to fix that right now)\r\n{code}\r\nwith fetch-size as 200, using DataStax Java driver. \r\n\r\nThis table contains about 20k rows. \r\n\r\n \r\n\r\nActually, the fix is quite simple, \r\n\r\n \r\n{code}\r\n--- a/src/java/org/apache/cassandra/service/pager/PagingState.java\r\n+++ b/src/java/org/apache/cassandra/service/pager/PagingState.java\r\n@@ -46,7 +46,7 @@ public class PagingState\r\n\r\npublic PagingState(ByteBuffer partitionKey, RowMark rowMark, int remaining, int remainingInPartition)\r\n {\r\n- this.partitionKey = partitionKey;\r\n+ this.partitionKey = partitionKey == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : partitionKey;\r\n this.rowMark = rowMark;\r\n this.remaining = remaining;\r\n this.remainingInPartition = remainingInPartition;\r\n{code}\r\n \r\n\r\n\"partitionKey == null ? ByteBufferUtil.EMPTY_BYTE_BUFFER : partitionKey;\" is in 2.2.6 and 2.2.8. But it was removed for some reason. \r\n\r\nThe interesting part is that, we have: \r\n{code}\r\npublic final ByteBuffer partitionKey; // Can be null for single partition queries.\r\n{code}\r\nIt seems \"partitionKey\" could be null.\r\n\r\nThanks a lot. \r\n\r\n \r\n\r\n \r\n\r\n \n\n---\n\nFrom: developer\n\nHere is my PR: [https://github.com/apache/cassandra/pull/225]\r\n\r\nCould anyone please have a look? Thanks a lot. ", "target_key": "CASSANDRA-10880"} {"cluster": "JIRA-CASSANDRA-c5429e9e9c65", "issue_id": "12630463", "key": "CASSANDRA-5215", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Ec2Snitch does not work properly\n\nWhen setting my endpoint_snitch to be Ec2Snitch, I get this:\n\n{CODE}\nERROR [main] 2013-02-01 22:57:20,742 DatabaseDescriptor.java (line 509) Fatal configuration error\norg.apache.cassandra.exceptions.ConfigurationException: Error instantiating snitch class 'org.apache.cassandra.locator.Ec2Snitch'.\n at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:475)\n at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:525)\n at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:338)\n at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:122)\n at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:151)\n at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:315)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n at java.lang.reflect.Method.invoke(Method.java:597)\n at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)\nCaused by: java.lang.reflect.InvocationTargetException\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:513)\n at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:457)\n ... 10 more\nCaused by: java.lang.ExceptionInInitializerError\n at org.apache.cassandra.locator.Ec2Snitch.(Ec2Snitch.java:65)\n ... 15 more\nCaused by: java.lang.RuntimeException: Unable to read cassandra-rackdc.properties\n at org.apache.cassandra.locator.SnitchProperties.(SnitchProperties.java:39)\n ... 16 more\nCaused by: java.lang.NullPointerException\n at java.util.Properties$LineReader.readLine(Properties.java:418)\n at java.util.Properties.load0(Properties.java:337)\n at java.util.Properties.load(Properties.java:325)\n at org.apache.cassandra.locator.SnitchProperties.(SnitchProperties.java:35)\n ... 16 more\n{CODE}\n\nMoving to SimpleSnitch allows the node to startup fine.", "query_30": "From: reporter\nSubject: Ec2Snitch does not work properly\n\nWhen setting my endpoint_snitch to be Ec2Snitch, I get this:\n\n{CODE}\nERROR [main] 2013-02-01 22:57:20,742 DatabaseDescriptor.java (line 509) Fatal configuration error\norg.apache.cassandra.exceptions.ConfigurationException: Error instantiating snitch class 'org.apache.cassandra.locator.Ec2Snitch'.\n at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:475)\n at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:525)\n at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:338)\n at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:122)\n at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:151)\n at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:315)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n at java.lang.reflect.Method.invoke(Method.java:597)\n at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)\nCaused by: java.lang.reflect.InvocationTargetException\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:513)\n at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:457)\n ... 10 more\nCaused by: java.lang.ExceptionInInitializerError\n at org.apache.cassandra.locator.Ec2Snitch.(Ec2Snitch.java:65)\n ... 15 more\nCaused by: java.lang.RuntimeException: Unable to read cassandra-rackdc.properties\n at org.apache.cassandra.locator.SnitchProperties.(SnitchProperties.java:39)\n ... 16 more\nCaused by: java.lang.NullPointerException\n at java.util.Properties$LineReader.readLine(Properties.java:418)\n at java.util.Properties.load0(Properties.java:337)\n at java.util.Properties.load(Properties.java:325)\n at org.apache.cassandra.locator.SnitchProperties.(SnitchProperties.java:35)\n ... 16 more\n{CODE}\n\nMoving to SimpleSnitch allows the node to startup fine.\n\n---\n\nFrom: developer\n\nis the file \"cassandra-rackdc.properties\" on the classpath? ", "query_60": "From: reporter\nSubject: Ec2Snitch does not work properly\n\nWhen setting my endpoint_snitch to be Ec2Snitch, I get this:\n\n{CODE}\nERROR [main] 2013-02-01 22:57:20,742 DatabaseDescriptor.java (line 509) Fatal configuration error\norg.apache.cassandra.exceptions.ConfigurationException: Error instantiating snitch class 'org.apache.cassandra.locator.Ec2Snitch'.\n at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:475)\n at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:525)\n at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:338)\n at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:122)\n at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:151)\n at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:315)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n at java.lang.reflect.Method.invoke(Method.java:597)\n at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:212)\nCaused by: java.lang.reflect.InvocationTargetException\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:513)\n at org.apache.cassandra.utils.FBUtilities.construct(FBUtilities.java:457)\n ... 10 more\nCaused by: java.lang.ExceptionInInitializerError\n at org.apache.cassandra.locator.Ec2Snitch.(Ec2Snitch.java:65)\n ... 15 more\nCaused by: java.lang.RuntimeException: Unable to read cassandra-rackdc.properties\n at org.apache.cassandra.locator.SnitchProperties.(SnitchProperties.java:39)\n ... 16 more\nCaused by: java.lang.NullPointerException\n at java.util.Properties$LineReader.readLine(Properties.java:418)\n at java.util.Properties.load0(Properties.java:337)\n at java.util.Properties.load(Properties.java:325)\n at org.apache.cassandra.locator.SnitchProperties.(SnitchProperties.java:35)\n ... 16 more\n{CODE}\n\nMoving to SimpleSnitch allows the node to startup fine.\n\n---\n\nFrom: developer\n\nis the file \"cassandra-rackdc.properties\" on the classpath? ", "target_key": "CASSANDRA-5212"} {"cluster": "JIRA-CASSANDRA-63efa8001612", "issue_id": "12703286", "key": "CASSANDRA-6918", "n_comments": 6, "progress_valid": {"0": true, "30": true, "60": true}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Compaction Assert: Incorrect Row Data Size\n\nI have four tables in a schema with Replication Factor: 6 (previously we set this to 3, but when we added more nodes we figured adding more replication to improve read time would help, this might have aggravated the issue).\n\ncreate table table_value_one (\n id timeuuid PRIMARY KEY,\n value_1 counter\n );\n \ncreate table table_value_two (\n id timeuuid PRIMARY KEY,\n value_2 counter\n );\n\ncreate table table_position_lookup (\n value_1 bigint,\n value_2 bigint,\n id timeuuid,\n PRIMARY KEY (id)\n ) WITH compaction={'class': 'LeveledCompactionStrategy'};\n\ncreate table sorted_table (\n row_key_index text,\n range bigint,\n sorted_value bigint,\n id timeuuid,\n extra_data list,\n PRIMARY KEY ((row_key_index, range), sorted_value, id)\n ) WITH CLUSTERING ORDER BY (sorted_value DESC) AND\n compaction={'class': 'LeveledCompactionStrategy'};\n\nThe application creates an object, and stores it in sorted_table based on a value position - for example, an object has a value_1 of 5500, and a value_2 of 4300.\n\nThere are rows which represent indices by which I can sort items based on these values in descending order. If I wish to see items with the highest # of value_1, I can create an index that stores them like so:\n\nrow_key_index = 'highest_value_1s'\n\nAdditionally, we shard each row by bucket ranges - which is simply the value_1 or value_2 / 1000. For example, our object above would be found in row_key_index = 'highest_value_1s' and range 5000, and also in row_key_index = 'highest_value_2s' with range 4300.\n\nThe true values of this object are stored in two counter tables, table_value_one and table_value_two. The current indexed position is stored in table_position_lookup.\n\nWe allow the application to modify value_one and value_two in the counter table indiscriminately. If we know the current values for these are dirty, we wait a tuned amount of time before we update the position in the sorted_table index. This creates 2 delete operations, and 2 write operations on the same table.\n\nThe issue is when we expand the number of write/delete operations on sorted_table, we see the following assert in the system log:\n\nERROR [CompactionExecutor:169] 2014-03-24 08:07:12,871 CassandraDaemon.java (line 191) Exception in thread Thread[CompactionExecutor:169,1,main]\njava.lang.AssertionError: incorrect row data size 77705872 written to /var/lib/cassandra/data/loadtest_1/sorted_table/loadtest_1-sorted_table-tmp-ic-165-Data.db; correct is 77800512\n at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)\n at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)\n at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)\n at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)\n at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)\n at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n at java.lang.Thread.run(Thread.java:724)\n\nEach object creates approximately ~500 unique row keys in sorted_table, and it possesses an extra_data field containing approximately 15 different bigint values.\n\nPreviously, our application was running Cassandra 1.2.10 and we did not see the assert when our sorted_table did not have the \"extra data list\". Also, we were writing around ~200 unique row keys, only containing the ID column.\n\nWe tried both leveled compaction and size tiered compaction and both cause the same assert - compaction fails to happen, and after about 100k object writes (creating 55 million rows, each having potentially as many as 100k items in a single column), we have ~ 2.4 GB of SSTables spread across 4840 files, and 691 SSTables:\n\n\t\tSSTable count: 691\n SSTables in each level: [685/4, 6, 0, 0, 0, 0, 0, 0, 0]\n Space used (live): 2244774352\n Space used (total): 2251159892\n SSTable Compression Ratio: 0.15101393198465862\n Number of Keys (estimate): 4704128\n Memtable Columns Count: 0\n Memtable Data Size: 0\n Memtable Switch Count: 264\n Read Count: 9204\n Read Latency: NaN ms.\n Write Count: 10151343\n Write Latency: NaN ms.\n Pending Tasks: 0\n Bloom Filter False Positives: 0\n Bloom Filter False Ratio: 0.00000\n Bloom Filter Space Used: 3500496\n Compacted row minimum size: 125\n Compacted row maximum size: 62479625\n Compacted row mean size: 1285302\n Average live cells per slice (last five minutes): 1001.0\n Average tombstones per slice (last five minutes): 8566.5\n\nSome mitigation strategies we have discussed include:\n* Breaking sorted_table into multiple column families to spread the # of writes between.\n* Increasing the coalescing time delay\n* Removing extra_data and paying the cost of another table look up for each item\n* Compressing extra_data into a blob\n* Reduce replication factor back down to 3 to reduce size pressure on SSTable.\n\nRunning nodetool -pr repair does not fix the issue. Running nodetool compact manually has not solved the issue as well. The asserts happen pretty frequently across all nodes of the cluster.", "query_30": "From: reporter\nSubject: Compaction Assert: Incorrect Row Data Size\n\nI have four tables in a schema with Replication Factor: 6 (previously we set this to 3, but when we added more nodes we figured adding more replication to improve read time would help, this might have aggravated the issue).\n\ncreate table table_value_one (\n id timeuuid PRIMARY KEY,\n value_1 counter\n );\n \ncreate table table_value_two (\n id timeuuid PRIMARY KEY,\n value_2 counter\n );\n\ncreate table table_position_lookup (\n value_1 bigint,\n value_2 bigint,\n id timeuuid,\n PRIMARY KEY (id)\n ) WITH compaction={'class': 'LeveledCompactionStrategy'};\n\ncreate table sorted_table (\n row_key_index text,\n range bigint,\n sorted_value bigint,\n id timeuuid,\n extra_data list,\n PRIMARY KEY ((row_key_index, range), sorted_value, id)\n ) WITH CLUSTERING ORDER BY (sorted_value DESC) AND\n compaction={'class': 'LeveledCompactionStrategy'};\n\nThe application creates an object, and stores it in sorted_table based on a value position - for example, an object has a value_1 of 5500, and a value_2 of 4300.\n\nThere are rows which represent indices by which I can sort items based on these values in descending order. If I wish to see items with the highest # of value_1, I can create an index that stores them like so:\n\nrow_key_index = 'highest_value_1s'\n\nAdditionally, we shard each row by bucket ranges - which is simply the value_1 or value_2 / 1000. For example, our object above would be found in row_key_index = 'highest_value_1s' and range 5000, and also in row_key_index = 'highest_value_2s' with range 4300.\n\nThe true values of this object are stored in two counter tables, table_value_one and table_value_two. The current indexed position is stored in table_position_lookup.\n\nWe allow the application to modify value_one and value_two in the counter table indiscriminately. If we know the current values for these are dirty, we wait a tuned amount of time before we update the position in the sorted_table index. This creates 2 delete operations, and 2 write operations on the same table.\n\nThe issue is when we expand the number of write/delete operations on sorted_table, we see the following assert in the system log:\n\nERROR [CompactionExecutor:169] 2014-03-24 08:07:12,871 CassandraDaemon.java (line 191) Exception in thread Thread[CompactionExecutor:169,1,main]\njava.lang.AssertionError: incorrect row data size 77705872 written to /var/lib/cassandra/data/loadtest_1/sorted_table/loadtest_1-sorted_table-tmp-ic-165-Data.db; correct is 77800512\n at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)\n at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)\n at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)\n at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)\n at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)\n at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n at java.lang.Thread.run(Thread.java:724)\n\nEach object creates approximately ~500 unique row keys in sorted_table, and it possesses an extra_data field containing approximately 15 different bigint values.\n\nPreviously, our application was running Cassandra 1.2.10 and we did not see the assert when our sorted_table did not have the \"extra data list\". Also, we were writing around ~200 unique row keys, only containing the ID column.\n\nWe tried both leveled compaction and size tiered compaction and both cause the same assert - compaction fails to happen, and after about 100k object writes (creating 55 million rows, each having potentially as many as 100k items in a single column), we have ~ 2.4 GB of SSTables spread across 4840 files, and 691 SSTables:\n\n\t\tSSTable count: 691\n SSTables in each level: [685/4, 6, 0, 0, 0, 0, 0, 0, 0]\n Space used (live): 2244774352\n Space used (total): 2251159892\n SSTable Compression Ratio: 0.15101393198465862\n Number of Keys (estimate): 4704128\n Memtable Columns Count: 0\n Memtable Data Size: 0\n Memtable Switch Count: 264\n Read Count: 9204\n Read Latency: NaN ms.\n Write Count: 10151343\n Write Latency: NaN ms.\n Pending Tasks: 0\n Bloom Filter False Positives: 0\n Bloom Filter False Ratio: 0.00000\n Bloom Filter Space Used: 3500496\n Compacted row minimum size: 125\n Compacted row maximum size: 62479625\n Compacted row mean size: 1285302\n Average live cells per slice (last five minutes): 1001.0\n Average tombstones per slice (last five minutes): 8566.5\n\nSome mitigation strategies we have discussed include:\n* Breaking sorted_table into multiple column families to spread the # of writes between.\n* Increasing the coalescing time delay\n* Removing extra_data and paying the cost of another table look up for each item\n* Compressing extra_data into a blob\n* Reduce replication factor back down to 3 to reduce size pressure on SSTable.\n\nRunning nodetool -pr repair does not fix the issue. Running nodetool compact manually has not solved the issue as well. The asserts happen pretty frequently across all nodes of the cluster.\n\n---\n\nFrom: developer\n\n[~iamaleksey] is this something that counters++ will fix or do you think it is more general than counters?\n\n---\n\nFrom: developer\n\n[~jbellis] I could be reading it wrong, but it seems like their issue is with the `sorted_table` table, and that one is counter-less.", "query_60": "From: reporter\nSubject: Compaction Assert: Incorrect Row Data Size\n\nI have four tables in a schema with Replication Factor: 6 (previously we set this to 3, but when we added more nodes we figured adding more replication to improve read time would help, this might have aggravated the issue).\n\ncreate table table_value_one (\n id timeuuid PRIMARY KEY,\n value_1 counter\n );\n \ncreate table table_value_two (\n id timeuuid PRIMARY KEY,\n value_2 counter\n );\n\ncreate table table_position_lookup (\n value_1 bigint,\n value_2 bigint,\n id timeuuid,\n PRIMARY KEY (id)\n ) WITH compaction={'class': 'LeveledCompactionStrategy'};\n\ncreate table sorted_table (\n row_key_index text,\n range bigint,\n sorted_value bigint,\n id timeuuid,\n extra_data list,\n PRIMARY KEY ((row_key_index, range), sorted_value, id)\n ) WITH CLUSTERING ORDER BY (sorted_value DESC) AND\n compaction={'class': 'LeveledCompactionStrategy'};\n\nThe application creates an object, and stores it in sorted_table based on a value position - for example, an object has a value_1 of 5500, and a value_2 of 4300.\n\nThere are rows which represent indices by which I can sort items based on these values in descending order. If I wish to see items with the highest # of value_1, I can create an index that stores them like so:\n\nrow_key_index = 'highest_value_1s'\n\nAdditionally, we shard each row by bucket ranges - which is simply the value_1 or value_2 / 1000. For example, our object above would be found in row_key_index = 'highest_value_1s' and range 5000, and also in row_key_index = 'highest_value_2s' with range 4300.\n\nThe true values of this object are stored in two counter tables, table_value_one and table_value_two. The current indexed position is stored in table_position_lookup.\n\nWe allow the application to modify value_one and value_two in the counter table indiscriminately. If we know the current values for these are dirty, we wait a tuned amount of time before we update the position in the sorted_table index. This creates 2 delete operations, and 2 write operations on the same table.\n\nThe issue is when we expand the number of write/delete operations on sorted_table, we see the following assert in the system log:\n\nERROR [CompactionExecutor:169] 2014-03-24 08:07:12,871 CassandraDaemon.java (line 191) Exception in thread Thread[CompactionExecutor:169,1,main]\njava.lang.AssertionError: incorrect row data size 77705872 written to /var/lib/cassandra/data/loadtest_1/sorted_table/loadtest_1-sorted_table-tmp-ic-165-Data.db; correct is 77800512\n at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)\n at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)\n at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)\n at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)\n at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)\n at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n at java.lang.Thread.run(Thread.java:724)\n\nEach object creates approximately ~500 unique row keys in sorted_table, and it possesses an extra_data field containing approximately 15 different bigint values.\n\nPreviously, our application was running Cassandra 1.2.10 and we did not see the assert when our sorted_table did not have the \"extra data list\". Also, we were writing around ~200 unique row keys, only containing the ID column.\n\nWe tried both leveled compaction and size tiered compaction and both cause the same assert - compaction fails to happen, and after about 100k object writes (creating 55 million rows, each having potentially as many as 100k items in a single column), we have ~ 2.4 GB of SSTables spread across 4840 files, and 691 SSTables:\n\n\t\tSSTable count: 691\n SSTables in each level: [685/4, 6, 0, 0, 0, 0, 0, 0, 0]\n Space used (live): 2244774352\n Space used (total): 2251159892\n SSTable Compression Ratio: 0.15101393198465862\n Number of Keys (estimate): 4704128\n Memtable Columns Count: 0\n Memtable Data Size: 0\n Memtable Switch Count: 264\n Read Count: 9204\n Read Latency: NaN ms.\n Write Count: 10151343\n Write Latency: NaN ms.\n Pending Tasks: 0\n Bloom Filter False Positives: 0\n Bloom Filter False Ratio: 0.00000\n Bloom Filter Space Used: 3500496\n Compacted row minimum size: 125\n Compacted row maximum size: 62479625\n Compacted row mean size: 1285302\n Average live cells per slice (last five minutes): 1001.0\n Average tombstones per slice (last five minutes): 8566.5\n\nSome mitigation strategies we have discussed include:\n* Breaking sorted_table into multiple column families to spread the # of writes between.\n* Increasing the coalescing time delay\n* Removing extra_data and paying the cost of another table look up for each item\n* Compressing extra_data into a blob\n* Reduce replication factor back down to 3 to reduce size pressure on SSTable.\n\nRunning nodetool -pr repair does not fix the issue. Running nodetool compact manually has not solved the issue as well. The asserts happen pretty frequently across all nodes of the cluster.\n\n---\n\nFrom: developer\n\n[~iamaleksey] is this something that counters++ will fix or do you think it is more general than counters?\n\n---\n\nFrom: developer\n\n[~jbellis] I could be reading it wrong, but it seems like their issue is with the `sorted_table` table, and that one is counter-less.\n\n---\n\nFrom: developer\n\n[~agoodrich] [~redpriest] does it log \"Compacting large row\" before the exception?\n\n---\n\nFrom: developer\n\nYes, this is a counter-less table that the exceptions occur on - [~jbellis] It depends on the node - here's an exception on node #2 in my cluster - I've seen it happen without (seemingly) a corresponding compaction large row. Here's an example where there is one directly above it:\n\nINFO [CompactionExecutor:144] 2014-03-24 07:50:33,240 CompactionController.java (line 156) Compacting large row loadtest_1/sorted_table:category1_globallist_item_4:0 (67157460 bytes) incrementally\nERROR [CompactionExecutor:144] 2014-03-24 07:50:42,471 CassandraDaemon.java (line 191) Exception in thread Thread[CompactionExecutor:144,1,main]\njava.lang.AssertionError: incorrect row data size 67156948 written to /var/lib/cassandra/data/loadtest_1/sorted_table/loadtest_1-sorted_table-tmp-ic-77-Data.db; correct is 67239030\n at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)\n at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:162)\n at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)\n at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)\n at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)\n at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n at java.util.concurrent.FutureTask.run(FutureTask.java:262)\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n at java.lang.Thread.run(Thread.java:724)\n", "target_key": "CASSANDRA-4180"} {"cluster": "JIRA-CASSANDRA-fa595c03c0ec", "issue_id": "12733783", "key": "CASSANDRA-7762", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: range query on secondary index times out (1 row, 1 node)\n\nSetup: \n- single cassandra node (on localhost)\n- in cassandra.yaml, listen_address: \"127.0.0.1\" and seeds: \"127.0.0.1\"\n\nHere is my test schema:\n\nCREATE TABLE foo (\n foo_name ascii,\n foo_shard bigint,\n int_val bigint,\n PRIMARY KEY ((foo_name, foo_shard))\n) WITH read_repair_chance=0.1;\n\nCREATE INDEX ON foo (int_val);\nCREATE INDEX ON foo (foo_name);\n\nI have inserted just a single row into this table:\ninsert into foo(foo_name, foo_shard, int_val) values('dave', 27, 100);\n\nThis query works fine:\n> select * from foo where foo_name='dave';\n\nBut when I run this query, I get an RPC timeout:\n> select * from foo where foo_name='dave' and int_val > 0 allow filtering;\nRequest did not complete within rpc_timeout.\n\nWith tracing enabled, here is the trace output (also attached):\nhttp://pastebin.com/raw.php?i=6XMEVUcQ\n\n(In short, everything looks fine to my untrained eye until 10s elapsed, at which time the following event is logged: \"Timed out; received 0 of 1 responses for range 257 of 257\")\n\nAlso note that if I *remove* the index on int_val (leaving just the index on foo_name), then the query works fine:\n\n> select * from foo where foo_name='dave' and int_val > 0 allow filtering;\n\n foo_name | foo_shard | int_val\n----------+-----------+---------\n dave | 27 | 100\n\n(1 rows)\n", "query_30": "From: reporter\nSubject: range query on secondary index times out (1 row, 1 node)\n\nSetup: \n- single cassandra node (on localhost)\n- in cassandra.yaml, listen_address: \"127.0.0.1\" and seeds: \"127.0.0.1\"\n\nHere is my test schema:\n\nCREATE TABLE foo (\n foo_name ascii,\n foo_shard bigint,\n int_val bigint,\n PRIMARY KEY ((foo_name, foo_shard))\n) WITH read_repair_chance=0.1;\n\nCREATE INDEX ON foo (int_val);\nCREATE INDEX ON foo (foo_name);\n\nI have inserted just a single row into this table:\ninsert into foo(foo_name, foo_shard, int_val) values('dave', 27, 100);\n\nThis query works fine:\n> select * from foo where foo_name='dave';\n\nBut when I run this query, I get an RPC timeout:\n> select * from foo where foo_name='dave' and int_val > 0 allow filtering;\nRequest did not complete within rpc_timeout.\n\nWith tracing enabled, here is the trace output (also attached):\nhttp://pastebin.com/raw.php?i=6XMEVUcQ\n\n(In short, everything looks fine to my untrained eye until 10s elapsed, at which time the following event is logged: \"Timed out; received 0 of 1 responses for range 257 of 257\")\n\nAlso note that if I *remove* the index on int_val (leaving just the index on foo_name), then the query works fine:\n\n> select * from foo where foo_name='dave' and int_val > 0 allow filtering;\n\n foo_name | foo_shard | int_val\n----------+-----------+---------\n dave | 27 | 100\n\n(1 rows)\n\n\n---\n\nFrom: developer\n\nI actually can reproduce the same issue. Mac OS X Mavericks, C* 2.0.9\n\nMy traces are quite similar, shorter\n\n{code:sql}\nTracing session: 9967bb90-2322-11e4-aad7-f75797a92abd\n activity | timestamp | source | source_elapsed\n-------------------------------------------------------------------------------------------------+--------------+-----------+----------------\n execute_cql3_query | 21:47:03,754 | 127.0.0.1 | 0\n Parsing select * from foo where foo_name='dave' and int_val > 0 allow filtering; | 21:47:03,755 | 127.0.0.1 | 1116\n Preparing statement | 21:47:03,756 | 127.0.0.1 | 1826\n Determining replicas to query | 21:47:03,758 | 127.0.0.1 | 4584\n No applicable indexes found | 21:47:03,769 | 127.0.0.1 | 14993\n Executing seq scan across 0 sstables for [min(-9223372036854775808), min(-9223372036854775808)] | 21:47:03,769 | 127.0.0.1 | 15048\n Read 1 live and 0 tombstoned cells | 21:47:03,769 | 127.0.0.1 | 15123\n Scanned 1 rows and matched 0 | 21:47:03,769 | 127.0.0.1 | 15215\n Timed out; received 0 of 1 responses for range 257 of 257 | 21:47:13,771 | 127.0.0.1 | 10017413\n Request complete | 21:47:13,775 | 127.0.0.1 | 10021748\n{code}\n\nThe duration goes through the roof for the \"Scanned 1 rows and matched 0\" step", "query_60": "From: reporter\nSubject: range query on secondary index times out (1 row, 1 node)\n\nSetup: \n- single cassandra node (on localhost)\n- in cassandra.yaml, listen_address: \"127.0.0.1\" and seeds: \"127.0.0.1\"\n\nHere is my test schema:\n\nCREATE TABLE foo (\n foo_name ascii,\n foo_shard bigint,\n int_val bigint,\n PRIMARY KEY ((foo_name, foo_shard))\n) WITH read_repair_chance=0.1;\n\nCREATE INDEX ON foo (int_val);\nCREATE INDEX ON foo (foo_name);\n\nI have inserted just a single row into this table:\ninsert into foo(foo_name, foo_shard, int_val) values('dave', 27, 100);\n\nThis query works fine:\n> select * from foo where foo_name='dave';\n\nBut when I run this query, I get an RPC timeout:\n> select * from foo where foo_name='dave' and int_val > 0 allow filtering;\nRequest did not complete within rpc_timeout.\n\nWith tracing enabled, here is the trace output (also attached):\nhttp://pastebin.com/raw.php?i=6XMEVUcQ\n\n(In short, everything looks fine to my untrained eye until 10s elapsed, at which time the following event is logged: \"Timed out; received 0 of 1 responses for range 257 of 257\")\n\nAlso note that if I *remove* the index on int_val (leaving just the index on foo_name), then the query works fine:\n\n> select * from foo where foo_name='dave' and int_val > 0 allow filtering;\n\n foo_name | foo_shard | int_val\n----------+-----------+---------\n dave | 27 | 100\n\n(1 rows)\n\n\n---\n\nFrom: developer\n\nI actually can reproduce the same issue. Mac OS X Mavericks, C* 2.0.9\n\nMy traces are quite similar, shorter\n\n{code:sql}\nTracing session: 9967bb90-2322-11e4-aad7-f75797a92abd\n activity | timestamp | source | source_elapsed\n-------------------------------------------------------------------------------------------------+--------------+-----------+----------------\n execute_cql3_query | 21:47:03,754 | 127.0.0.1 | 0\n Parsing select * from foo where foo_name='dave' and int_val > 0 allow filtering; | 21:47:03,755 | 127.0.0.1 | 1116\n Preparing statement | 21:47:03,756 | 127.0.0.1 | 1826\n Determining replicas to query | 21:47:03,758 | 127.0.0.1 | 4584\n No applicable indexes found | 21:47:03,769 | 127.0.0.1 | 14993\n Executing seq scan across 0 sstables for [min(-9223372036854775808), min(-9223372036854775808)] | 21:47:03,769 | 127.0.0.1 | 15048\n Read 1 live and 0 tombstoned cells | 21:47:03,769 | 127.0.0.1 | 15123\n Scanned 1 rows and matched 0 | 21:47:03,769 | 127.0.0.1 | 15215\n Timed out; received 0 of 1 responses for range 257 of 257 | 21:47:13,771 | 127.0.0.1 | 10017413\n Request complete | 21:47:13,775 | 127.0.0.1 | 10021748\n{code}\n\nThe duration goes through the roof for the \"Scanned 1 rows and matched 0\" step", "target_key": "CASSANDRA-6612"} {"cluster": "JIRA-CASSANDRA-a80341721d34", "issue_id": "12761130", "key": "CASSANDRA-8461", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: java.lang.AssertionError when running select queries\n\nI have a column family with following schema.\n\nCREATE TABLE corpus.trigram_category_ordered_frequency (\n id bigint,\n word1 varchar,\n word2 varchar,\n word3 varchar,\n category varchar,\n frequency int,\n PRIMARY KEY(category,frequency,word1,word2,word3)\n);\n\nWhen I run \n\n select word1,word2,word3 from corpus.trigram_category_ordered_frequency where category IN ('N','A','C','S','G') order by frequency DESC LIMIT 10;\n\nI am getting error saying\n\n\n\nBut when I ran \n\nselect * from corpus.trigram_category_ordered_frequency where category IN ('N','A','C','S','G') order by frequency DESC LIMIT 10;\n\nit works without any error.\n\nsystem log for this error is as follows.\n\n{code}\nERROR [SharedPool-Worker-1] 2014-12-11 20:42:20,152 Message.java:538 - Unexpected exception during request; channel = [id: 0xea57d8b6, /127.0.0.1:35624 => /127.0.0.1:9042]\njava.lang.AssertionError: null\n\tat org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:333) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1227) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1161) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:290) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:267) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:215) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_72]\n\tat org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]{code}", "query_30": "From: reporter\nSubject: java.lang.AssertionError when running select queries\n\nI have a column family with following schema.\n\nCREATE TABLE corpus.trigram_category_ordered_frequency (\n id bigint,\n word1 varchar,\n word2 varchar,\n word3 varchar,\n category varchar,\n frequency int,\n PRIMARY KEY(category,frequency,word1,word2,word3)\n);\n\nWhen I run \n\n select word1,word2,word3 from corpus.trigram_category_ordered_frequency where category IN ('N','A','C','S','G') order by frequency DESC LIMIT 10;\n\nI am getting error saying\n\n\n\nBut when I ran \n\nselect * from corpus.trigram_category_ordered_frequency where category IN ('N','A','C','S','G') order by frequency DESC LIMIT 10;\n\nit works without any error.\n\nsystem log for this error is as follows.\n\n{code}\nERROR [SharedPool-Worker-1] 2014-12-11 20:42:20,152 Message.java:538 - Unexpected exception during request; channel = [id: 0xea57d8b6, /127.0.0.1:35624 => /127.0.0.1:9042]\njava.lang.AssertionError: null\n\tat org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:333) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1227) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1161) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:290) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:267) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:215) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_72]\n\tat org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]{code}\n\n---\n\nFrom: developer\n\n[~thobbs], I will verify that this was not fixed by other work.", "query_60": "From: reporter\nSubject: java.lang.AssertionError when running select queries\n\nI have a column family with following schema.\n\nCREATE TABLE corpus.trigram_category_ordered_frequency (\n id bigint,\n word1 varchar,\n word2 varchar,\n word3 varchar,\n category varchar,\n frequency int,\n PRIMARY KEY(category,frequency,word1,word2,word3)\n);\n\nWhen I run \n\n select word1,word2,word3 from corpus.trigram_category_ordered_frequency where category IN ('N','A','C','S','G') order by frequency DESC LIMIT 10;\n\nI am getting error saying\n\n\n\nBut when I ran \n\nselect * from corpus.trigram_category_ordered_frequency where category IN ('N','A','C','S','G') order by frequency DESC LIMIT 10;\n\nit works without any error.\n\nsystem log for this error is as follows.\n\n{code}\nERROR [SharedPool-Worker-1] 2014-12-11 20:42:20,152 Message.java:538 - Unexpected exception during request; channel = [id: 0xea57d8b6, /127.0.0.1:35624 => /127.0.0.1:9042]\njava.lang.AssertionError: null\n\tat org.apache.cassandra.cql3.ResultSet.addRow(ResultSet.java:63) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.Selection$ResultSetBuilder.newRow(Selection.java:333) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.processColumnFamily(SelectStatement.java:1227) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:1161) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:290) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:267) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:215) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:64) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119) ~[apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_72]\n\tat org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2]\n\tat java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]{code}\n\n---\n\nFrom: developer\n\n[~thobbs], I will verify that this was not fixed by other work.", "target_key": "CASSANDRA-8286"} {"cluster": "JIRA-CASSANDRA-a3ad97e7b7f2", "issue_id": "12785340", "key": "CASSANDRA-9033", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: Upgrading from 2.1.1 to 2.1.3 with LCS and many sstable files makes nodes unresponsive\n\nWe have an Event Log table using LCS that has grown fast. There are more than 100K sstable files that are around 1KB. Increasing compactors and adjusting compaction throttling upward doesn't make a difference. It has been running great though until we upgraded to 2.1.3. Those nodes needed more RAM for the stack (12 GB) to even have a prayer of responding to queries. They bog down and become unresponsive. There are no GC messages that I can see, and no compaction either. \n\nThe only work-around I have found is to decommission, blow away the big CF and rejoin. That happens in about 20 minutes and everything is freaking happy again. The size of the files is more like what I'd expect as well. \n\nOur schema: \n\n{code}\ncqlsh> describe columnfamily data.stories\n\nCREATE TABLE data.stories (\n id timeuuid PRIMARY KEY,\n action_data timeuuid,\n action_name text,\n app_id timeuuid,\n app_instance_id timeuuid,\n data map,\n objects set,\n time_stamp timestamp,\n user_id timeuuid\n) WITH bloom_filter_fp_chance = 0.01\n AND caching = '{\"keys\":\"ALL\", \"rows_per_partition\":\"NONE\"}'\n AND comment = 'Stories represent the timeline and are placed in the dashboard for the brand manager to see'\n AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}\n AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}\n AND dclocal_read_repair_chance = 0.1\n AND default_time_to_live = 0\n AND gc_grace_seconds = 864000\n AND max_index_interval = 2048\n AND memtable_flush_period_in_ms = 0\n AND min_index_interval = 128\n AND read_repair_chance = 0.0\n AND speculative_retry = '99.0PERCENTILE';\n\ncqlsh> \n{code}\n\nThere were no log entries that stood out. It pretty much consisted of \"x is down\" \"x is up\" repeated ad infinitum. I have attached the zipped system.log that has the situation after the upgrade and then after I stopped, removed system, system_traces, OpsCenter, and data/stories-/* and restarted. \n\nIt has rejoined the cluster now and is busy read-repairing to recover its data.\n\nOn another note, we see a lot of this during repair now (on all the nodes): \n\n{code}\nERROR [AntiEntropySessions:5] 2015-03-24 20:03:10,207 RepairSession.java:303 - [repair #c5043c40-d260-11e4-a2f2-8bb3e2bbdb35] session completed with the following error\njava.io.IOException: Failed during snapshot creation.\n at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\nERROR [AntiEntropySessions:5] 2015-03-24 20:03:10,208 CassandraDaemon.java:167 - Exception in thread Thread[AntiEntropySessions:5,5,RMI Runtime]\njava.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.\n at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_55]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\nCaused by: java.io.IOException: Failed during snapshot creation.\n at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]\n ... 3 common frames omitted\nERROR [RepairJobTask:2] 2015-03-24 20:03:20,227 RepairJob.java:145 - Error occurred during snapshot phase\njava.lang.RuntimeException: Could not create snapshot at /10.0.2.144\n at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:349) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_55]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\n{code} \n\nI am thinking that this means that my work-around for blowing away and rebuilding the CF may not be working anymore. I don't know of another way to force LCS compaction. The node doesn't ever seem to recover enough to compact on its own.", "query_30": "From: reporter\nSubject: Upgrading from 2.1.1 to 2.1.3 with LCS and many sstable files makes nodes unresponsive\n\nWe have an Event Log table using LCS that has grown fast. There are more than 100K sstable files that are around 1KB. Increasing compactors and adjusting compaction throttling upward doesn't make a difference. It has been running great though until we upgraded to 2.1.3. Those nodes needed more RAM for the stack (12 GB) to even have a prayer of responding to queries. They bog down and become unresponsive. There are no GC messages that I can see, and no compaction either. \n\nThe only work-around I have found is to decommission, blow away the big CF and rejoin. That happens in about 20 minutes and everything is freaking happy again. The size of the files is more like what I'd expect as well. \n\nOur schema: \n\n{code}\ncqlsh> describe columnfamily data.stories\n\nCREATE TABLE data.stories (\n id timeuuid PRIMARY KEY,\n action_data timeuuid,\n action_name text,\n app_id timeuuid,\n app_instance_id timeuuid,\n data map,\n objects set,\n time_stamp timestamp,\n user_id timeuuid\n) WITH bloom_filter_fp_chance = 0.01\n AND caching = '{\"keys\":\"ALL\", \"rows_per_partition\":\"NONE\"}'\n AND comment = 'Stories represent the timeline and are placed in the dashboard for the brand manager to see'\n AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}\n AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}\n AND dclocal_read_repair_chance = 0.1\n AND default_time_to_live = 0\n AND gc_grace_seconds = 864000\n AND max_index_interval = 2048\n AND memtable_flush_period_in_ms = 0\n AND min_index_interval = 128\n AND read_repair_chance = 0.0\n AND speculative_retry = '99.0PERCENTILE';\n\ncqlsh> \n{code}\n\nThere were no log entries that stood out. It pretty much consisted of \"x is down\" \"x is up\" repeated ad infinitum. I have attached the zipped system.log that has the situation after the upgrade and then after I stopped, removed system, system_traces, OpsCenter, and data/stories-/* and restarted. \n\nIt has rejoined the cluster now and is busy read-repairing to recover its data.\n\nOn another note, we see a lot of this during repair now (on all the nodes): \n\n{code}\nERROR [AntiEntropySessions:5] 2015-03-24 20:03:10,207 RepairSession.java:303 - [repair #c5043c40-d260-11e4-a2f2-8bb3e2bbdb35] session completed with the following error\njava.io.IOException: Failed during snapshot creation.\n at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\nERROR [AntiEntropySessions:5] 2015-03-24 20:03:10,208 CassandraDaemon.java:167 - Exception in thread Thread[AntiEntropySessions:5,5,RMI Runtime]\njava.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.\n at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_55]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\nCaused by: java.io.IOException: Failed during snapshot creation.\n at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]\n ... 3 common frames omitted\nERROR [RepairJobTask:2] 2015-03-24 20:03:20,227 RepairJob.java:145 - Error occurred during snapshot phase\njava.lang.RuntimeException: Could not create snapshot at /10.0.2.144\n at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:349) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_55]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\n{code} \n\nI am thinking that this means that my work-around for blowing away and rebuilding the CF may not be working anymore. I don't know of another way to force LCS compaction. The node doesn't ever seem to recover enough to compact on its own.\n\n---\n\nFrom: developer\n\nLowering prio as the actual problem is that you have that many tiny files on your node. Question is how you ended up with that many files.\n\nDid you run repairs prior to the number of files exploded?\n\nDo you have graphs over how many files you have on the node? Is there a gradual increase over time or did it happen over night?", "query_60": "From: reporter\nSubject: Upgrading from 2.1.1 to 2.1.3 with LCS and many sstable files makes nodes unresponsive\n\nWe have an Event Log table using LCS that has grown fast. There are more than 100K sstable files that are around 1KB. Increasing compactors and adjusting compaction throttling upward doesn't make a difference. It has been running great though until we upgraded to 2.1.3. Those nodes needed more RAM for the stack (12 GB) to even have a prayer of responding to queries. They bog down and become unresponsive. There are no GC messages that I can see, and no compaction either. \n\nThe only work-around I have found is to decommission, blow away the big CF and rejoin. That happens in about 20 minutes and everything is freaking happy again. The size of the files is more like what I'd expect as well. \n\nOur schema: \n\n{code}\ncqlsh> describe columnfamily data.stories\n\nCREATE TABLE data.stories (\n id timeuuid PRIMARY KEY,\n action_data timeuuid,\n action_name text,\n app_id timeuuid,\n app_instance_id timeuuid,\n data map,\n objects set,\n time_stamp timestamp,\n user_id timeuuid\n) WITH bloom_filter_fp_chance = 0.01\n AND caching = '{\"keys\":\"ALL\", \"rows_per_partition\":\"NONE\"}'\n AND comment = 'Stories represent the timeline and are placed in the dashboard for the brand manager to see'\n AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'}\n AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}\n AND dclocal_read_repair_chance = 0.1\n AND default_time_to_live = 0\n AND gc_grace_seconds = 864000\n AND max_index_interval = 2048\n AND memtable_flush_period_in_ms = 0\n AND min_index_interval = 128\n AND read_repair_chance = 0.0\n AND speculative_retry = '99.0PERCENTILE';\n\ncqlsh> \n{code}\n\nThere were no log entries that stood out. It pretty much consisted of \"x is down\" \"x is up\" repeated ad infinitum. I have attached the zipped system.log that has the situation after the upgrade and then after I stopped, removed system, system_traces, OpsCenter, and data/stories-/* and restarted. \n\nIt has rejoined the cluster now and is busy read-repairing to recover its data.\n\nOn another note, we see a lot of this during repair now (on all the nodes): \n\n{code}\nERROR [AntiEntropySessions:5] 2015-03-24 20:03:10,207 RepairSession.java:303 - [repair #c5043c40-d260-11e4-a2f2-8bb3e2bbdb35] session completed with the following error\njava.io.IOException: Failed during snapshot creation.\n at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\nERROR [AntiEntropySessions:5] 2015-03-24 20:03:10,208 CassandraDaemon.java:167 - Exception in thread Thread[AntiEntropySessions:5,5,RMI Runtime]\njava.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.\n at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]\n at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_55]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\nCaused by: java.io.IOException: Failed during snapshot creation.\n at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:146) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]\n ... 3 common frames omitted\nERROR [RepairJobTask:2] 2015-03-24 20:03:20,227 RepairJob.java:145 - Error occurred during snapshot phase\njava.lang.RuntimeException: Could not create snapshot at /10.0.2.144\n at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:349) ~[apache-cassandra-2.1.3.jar:2.1.3]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_55]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]\n{code} \n\nI am thinking that this means that my work-around for blowing away and rebuilding the CF may not be working anymore. I don't know of another way to force LCS compaction. The node doesn't ever seem to recover enough to compact on its own.\n\n---\n\nFrom: developer\n\nLowering prio as the actual problem is that you have that many tiny files on your node. Question is how you ended up with that many files.\n\nDid you run repairs prior to the number of files exploded?\n\nDo you have graphs over how many files you have on the node? Is there a gradual increase over time or did it happen over night?\n\n---\n\nFrom: developer\n\nIf we run 2.1.3 our cluster *will* fail. We have started with upgraded data and we have started with an empty key store. No matter what, the number of files for all of our LCS tables will grow until the machines collapse. That seems like a blocker to me.\n\n* I don't have a graph of the number of files. \n* We have observed this with nodes after repair or during the day when no repairs have been run.\n* Repair doesn't seem to actually work on 2.1.3. Downgrading back to 2.1.1 restored the nodes' performance in every case and repair actually worked. After a few hours, everything was back to normal.\n\nSome more observations -\n\n* There are something like 50K to 147K files on the nodes for this table right now. C* 2.1.1 works with them. \n* There is plenty of bandwidth for compactions. No compactions are pending.\n* Downgrading to 2.1.1 from 2.1.3 caused all sorts of compactions and reduced the number of files down to under 100K (the count was so high I couldn't count them before)\n* Upgrading to 2.1.3 created issues immediately on some nodes, but other nodes performed fine until suddenly they couldn't.\n* We ran into issues on this cluster after successfully upgrading to 2.1.3. This is all log data for our app - it is the product of many storm machines humming away on data and we can rebuild it (although not without cost), so the first time we had trouble, I just wiped the disks and started over with 2.1.3. Within a week we had too many files and nodes were failing.\n* We never had a complete 2.1.3 cluster. We never made it through the upgrade before things went to shit, so it may be some interaction between 2.1.1 and 2.1.3 (seems unlikely). \n\nBottom line, if we use LCS with 2.1.3 we will fail.\n\nI have experimented ad nauseam with the settings. I have added compactors, increased compaction throughput, reduced it, added RAM to Heap, added off-heap RAM for memtables, trickled-fsync on or off. This will happen for us every time. \n\n# Are there settings I should look at specifically?\n# I will start a task to log file count for the table on 2.1.1 on all of the nodes. I don't expect it to grow.\n# Is there a way to force these files to be rolled up / compacted? \n", "target_key": "CASSANDRA-8860"} {"cluster": "JIRA-CASSANDRA-d3ad58e022d0", "issue_id": "12822428", "key": "CASSANDRA-9216", "n_comments": 5, "progress_valid": {"0": true, "30": true, "60": true}, "project": "CASSANDRA", "query_0": "From: reporter\nSubject: NullPointerException (NPE) during Compaction Cache Serialization\n\nIn case this hasn't been reported (I looked but did not see it), a null pointer exception is occurring during compaction. The stack track is as follows:\n{code}\nERROR [CompactionExecutor:50] 2015-04-20 13:42:43,827 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:50,1,main]\njava.lang.NullPointerException: null\n at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:274) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1152) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_75]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]{code}", "query_30": "From: reporter\nSubject: NullPointerException (NPE) during Compaction Cache Serialization\n\nIn case this hasn't been reported (I looked but did not see it), a null pointer exception is occurring during compaction. The stack track is as follows:\n{code}\nERROR [CompactionExecutor:50] 2015-04-20 13:42:43,827 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:50,1,main]\njava.lang.NullPointerException: null\n at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:274) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1152) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_75]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]{code}\n\n---\n\nFrom: developer\n\nCould you possibly attach the entire system.log file? Do you know what table this was compacting when the exception occurred? Could you give us the schema for that table, especially the compaction strategy/options?\n\n---\n\nFrom: developer\n\nUnfortunately, the system log was deleted as the /var/log partition filled up when the database fell over. I only got this Exception because of another process monitoring the logs for exceptions and sending them off board. I will be monitoring the system directly so if the problem occurs again, I will capture it.", "query_60": "From: reporter\nSubject: NullPointerException (NPE) during Compaction Cache Serialization\n\nIn case this hasn't been reported (I looked but did not see it), a null pointer exception is occurring during compaction. The stack track is as follows:\n{code}\nERROR [CompactionExecutor:50] 2015-04-20 13:42:43,827 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:50,1,main]\njava.lang.NullPointerException: null\n at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:475) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.service.CacheService$KeyCacheSerializer.serialize(CacheService.java:463) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:274) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at org.apache.cassandra.db.compaction.CompactionManager$11.run(CompactionManager.java:1152) ~[apache-cassandra-2.1.4.jar:2.1.4]\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_75]\n at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_75]\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_75]\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]\n at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]{code}\n\n---\n\nFrom: developer\n\nCould you possibly attach the entire system.log file? Do you know what table this was compacting when the exception occurred? Could you give us the schema for that table, especially the compaction strategy/options?\n\n---\n\nFrom: developer\n\nUnfortunately, the system log was deleted as the /var/log partition filled up when the database fell over. I only got this Exception because of another process monitoring the logs for exceptions and sending them off board. I will be monitoring the system directly so if the problem occurs again, I will capture it.\n\n---\n\nFrom: developer\n\nHave you run into this again? It may just have been a corrupt keycache file.", "target_key": "CASSANDRA-8067"} {"cluster": "JIRA-HADOOP-00eb2e5f269e", "issue_id": "12787384", "key": "HADOOP-11784", "n_comments": 3, "progress_valid": {"0": true, "30": true, "60": true}, "project": "HADOOP", "query_0": "From: reporter\nSubject: failed to locate Winutils for win 32 platform\n\nDuring the execution of Map reduce example first I got the error telling \nERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path\n\nSo I downloaded a version and update the bin folder again executing the same command I am getting error like \n\nERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path\njava.io.IOException: Could not locate executable C:\\hadoop-2.6.0\\bin\\winutils.exe in the Hadoop binaries.\n at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)\n at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)\n at org.apache.hadoop.util.Shell.(Shell.java:363)\n at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)\n at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)\n at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)\n at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)\n at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)\n at org.apache.hadoop.examples.Grep.main(Grep.java:101)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)\n at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)\n at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n at org.apache.hadoop.util.RunJar.main(RunJar.java:136)\n15/04/01 19:12:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where ap\nlicable\n15/04/01 19:12:52 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id\n15/04/01 19:12:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=\njava.lang.NullPointerException\n at java.lang.ProcessBuilder.start(ProcessBuilder.java:1011)\n at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)\n at org.apache.hadoop.util.Shell.run(Shell.java:455)\n at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)\n at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)\n at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)\n at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)\n at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444)\n at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:293)\n at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)\n at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)\n at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)\n at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)\n at java.security.AccessController.doPrivileged(Native Method)\n at javax.security.auth.Subject.doAs(Subject.java:415)\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\n at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)\n at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)\n at org.apache.hadoop.examples.Grep.run(Grep.java:77)\n at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)\n at org.apache.hadoop.examples.Grep.main(Grep.java:101)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)\n at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)\n at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n at org.apache.hadoop.util.RunJar.main(RunJar.java:136)\n\nPlease let me know if you have patch for the solution.\n", "query_30": "From: reporter\nSubject: failed to locate Winutils for win 32 platform\n\nDuring the execution of Map reduce example first I got the error telling \nERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path\n\nSo I downloaded a version and update the bin folder again executing the same command I am getting error like \n\nERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path\njava.io.IOException: Could not locate executable C:\\hadoop-2.6.0\\bin\\winutils.exe in the Hadoop binaries.\n at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)\n at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)\n at org.apache.hadoop.util.Shell.(Shell.java:363)\n at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)\n at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)\n at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)\n at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)\n at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)\n at org.apache.hadoop.examples.Grep.main(Grep.java:101)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)\n at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)\n at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n at org.apache.hadoop.util.RunJar.main(RunJar.java:136)\n15/04/01 19:12:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where ap\nlicable\n15/04/01 19:12:52 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id\n15/04/01 19:12:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=\njava.lang.NullPointerException\n at java.lang.ProcessBuilder.start(ProcessBuilder.java:1011)\n at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)\n at org.apache.hadoop.util.Shell.run(Shell.java:455)\n at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)\n at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)\n at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)\n at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)\n at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444)\n at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:293)\n at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)\n at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)\n at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)\n at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)\n at java.security.AccessController.doPrivileged(Native Method)\n at javax.security.auth.Subject.doAs(Subject.java:415)\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\n at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)\n at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)\n at org.apache.hadoop.examples.Grep.run(Grep.java:77)\n at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)\n at org.apache.hadoop.examples.Grep.main(Grep.java:101)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)\n at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)\n at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n at org.apache.hadoop.util.RunJar.main(RunJar.java:136)\n\nPlease let me know if you have patch for the solution.\n\n\n---\n\nFrom: developer\n\nDid you try to set HADOOP_HOME=C:\\hadoop-2.6.0?", "query_60": "From: reporter\nSubject: failed to locate Winutils for win 32 platform\n\nDuring the execution of Map reduce example first I got the error telling \nERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path\n\nSo I downloaded a version and update the bin folder again executing the same command I am getting error like \n\nERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path\njava.io.IOException: Could not locate executable C:\\hadoop-2.6.0\\bin\\winutils.exe in the Hadoop binaries.\n at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)\n at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)\n at org.apache.hadoop.util.Shell.(Shell.java:363)\n at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)\n at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)\n at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170)\n at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153)\n at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)\n at org.apache.hadoop.examples.Grep.main(Grep.java:101)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)\n at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)\n at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n at org.apache.hadoop.util.RunJar.main(RunJar.java:136)\n15/04/01 19:12:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where ap\nlicable\n15/04/01 19:12:52 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id\n15/04/01 19:12:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=\njava.lang.NullPointerException\n at java.lang.ProcessBuilder.start(ProcessBuilder.java:1011)\n at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)\n at org.apache.hadoop.util.Shell.run(Shell.java:455)\n at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)\n at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)\n at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)\n at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656)\n at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444)\n at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:293)\n at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)\n at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:437)\n at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)\n at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)\n at java.security.AccessController.doPrivileged(Native Method)\n at javax.security.auth.Subject.doAs(Subject.java:415)\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)\n at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)\n at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)\n at org.apache.hadoop.examples.Grep.run(Grep.java:77)\n at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)\n at org.apache.hadoop.examples.Grep.main(Grep.java:101)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)\n at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)\n at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:601)\n at org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n at org.apache.hadoop.util.RunJar.main(RunJar.java:136)\n\nPlease let me know if you have patch for the solution.\n\n\n---\n\nFrom: developer\n\nDid you try to set HADOOP_HOME=C:\\hadoop-2.6.0?\n\n---\n\nFrom: developer\n\nyes i have done", "target_key": "HADOOP-9922"} {"cluster": "JIRA-HADOOP-14bfc47fcbf0", "issue_id": "13060601", "key": "HADOOP-14262", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "HADOOP", "query_0": "From: reporter\nSubject: rpcTimeOut is not set up correctly in Client thus client doesn't time out\n\nNameNodeProxies.createNNProxyWithClientProtocol does\n\n{code}\n ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(\n ClientNamenodeProtocolPB.class, version, address, ugi, conf,\n NetUtils.getDefaultSocketFactory(conf),\n org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy,\n fallbackToSimpleAuth).getProxy();\n{code}\nwhich calls Client.getTimeOut(conf) to get timeout value. \n\nClient.getTimeOut(conf) doesn't consider IPC_CLIENT_RPC_TIMEOUT_KEY right now. Thus rpcTimeOut doesn't take effect for relevant RPC calls, and they hang!\n\nFor example, receiveRpcResponse blocked forever at:\n{code}\nThread 16127: (state = BLOCKED) \n - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled frame) \n - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 (Compiled frame) \n - org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) @bci=5, line=57 (Compiled frame)\n - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=35, line=142 (Compiled frame)\n - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=161 (Compiled frame)\n - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=131 (Compiled frame) \n - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame) \n - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=521 (Compiled frame)\n - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame) \n - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame) \n - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, line=1081 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled frame) \n{code}\n\nFiling this jira to fix it.\n", "query_30": "From: reporter\nSubject: rpcTimeOut is not set up correctly in Client thus client doesn't time out\n\nNameNodeProxies.createNNProxyWithClientProtocol does\n\n{code}\n ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(\n ClientNamenodeProtocolPB.class, version, address, ugi, conf,\n NetUtils.getDefaultSocketFactory(conf),\n org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy,\n fallbackToSimpleAuth).getProxy();\n{code}\nwhich calls Client.getTimeOut(conf) to get timeout value. \n\nClient.getTimeOut(conf) doesn't consider IPC_CLIENT_RPC_TIMEOUT_KEY right now. Thus rpcTimeOut doesn't take effect for relevant RPC calls, and they hang!\n\nFor example, receiveRpcResponse blocked forever at:\n{code}\nThread 16127: (state = BLOCKED) \n - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled frame) \n - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 (Compiled frame) \n - org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) @bci=5, line=57 (Compiled frame)\n - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=35, line=142 (Compiled frame)\n - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=161 (Compiled frame)\n - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=131 (Compiled frame) \n - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame) \n - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=521 (Compiled frame)\n - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame) \n - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame) \n - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, line=1081 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled frame) \n{code}\n\nFiling this jira to fix it.\n\n\n---\n\nFrom: developer\n\nClient.getTimeOut() is calling Client.getRpcTimeout(), right?", "query_60": "From: reporter\nSubject: rpcTimeOut is not set up correctly in Client thus client doesn't time out\n\nNameNodeProxies.createNNProxyWithClientProtocol does\n\n{code}\n ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(\n ClientNamenodeProtocolPB.class, version, address, ugi, conf,\n NetUtils.getDefaultSocketFactory(conf),\n org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy,\n fallbackToSimpleAuth).getProxy();\n{code}\nwhich calls Client.getTimeOut(conf) to get timeout value. \n\nClient.getTimeOut(conf) doesn't consider IPC_CLIENT_RPC_TIMEOUT_KEY right now. Thus rpcTimeOut doesn't take effect for relevant RPC calls, and they hang!\n\nFor example, receiveRpcResponse blocked forever at:\n{code}\nThread 16127: (state = BLOCKED) \n - sun.nio.ch.SocketChannelImpl.readerCleanup() @bci=6, line=279 (Compiled frame) \n - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=205, line=390 (Compiled frame) \n - org.apache.hadoop.net.SocketInputStream$Reader.performIO(java.nio.ByteBuffer) @bci=5, line=57 (Compiled frame)\n - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer, int) @bci=35, line=142 (Compiled frame)\n - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer) @bci=6, line=161 (Compiled frame)\n - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7, line=131 (Compiled frame) \n - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame) \n - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=133 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) @bci=4, line=521 (Compiled frame)\n - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame) \n - java.io.BufferedInputStream.read() @bci=12, line=265 (Compiled frame) \n - java.io.DataInputStream.readInt() @bci=4, line=387 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() @bci=19, line=1081 (Compiled frame) \n - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=976 (Compiled frame) \n{code}\n\nFiling this jira to fix it.\n\n\n---\n\nFrom: developer\n\nClient.getTimeOut() is calling Client.getRpcTimeout(), right?", "target_key": "HADOOP-12672"} {"cluster": "JIRA-HADOOP-6c3457425f28", "issue_id": "13115412", "key": "HADOOP-15011", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "HADOOP", "query_0": "From: reporter\nSubject: Getting file not found exception while using distcp with s3a\n\nI'm using the distcp option to copy the huge files from Hadoop to S3. Sometimes i'm getting the below error,\r\n\r\n*Command:* (Copying 378 GB data)\r\n\r\n_hadoop distcp -D HADOOP_OPTS=-Xmx12g -D HADOOP_CLIENT_OPTS='-Xmx12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled' -D 'mapreduce.map.memory.mb=12288' -D 'mapreduce.map.java.opts=-Xmx10g' -D 'mapreduce.reduce.memory.mb=12288' -D 'mapreduce.reduce.java.opts=-Xmx10g' '-Dfs.s3a.proxy.host=edhmgrn-prod.cloud.capitalone.com' '-Dfs.s3a.proxy.port=8088' '-Dfs.s3a.access.key=XXXXXXX' '-Dfs.s3a.secret.key=XXXXXXX' '-Dfs.s3a.connection.timeout=180000' '-Dfs.s3a.attempts.maximum=5' '-Dfs.s3a.fast.upload=true' '-Dfs.s3a.fast.upload.buffer=array' '-Dfs.s3a.fast.upload.active.blocks=50' '-Dfs.s3a.multipart.size=262144000' '-Dfs.s3a.threads.max=500' '-Dfs.s3a.threads.keepalivetime=600' '-Dfs.s3a.server-side-encryption-algorithm=AES256' -bandwidth 3072 -strategy dynamic -m 220 -numListstatusThreads 30 /src/ s3a://bucket/dest\r\n_\r\n17/11/01 12:23:27 INFO mapreduce.Job: Task Id : attempt_1497120915913_2792335_m_000165_0, Status : FAILED\r\nError: java.io.FileNotFoundException: No such file or directory: s3a://bucketname/filename\r\n\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1132)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:78)\r\n at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:197)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:256)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)\r\n at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)\r\n at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\r\n at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)\r\n at java.security.AccessController.doPrivileged(Native Method)\r\n at javax.security.auth.Subject.doAs(Subject.java:422)\r\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)\r\n at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\r\n\r\n17/11/01 12:28:32 INFO mapreduce.Job: Task Id : attempt_1497120915913_2792335_m_000010_0, Status : FAILED\r\nError: java.io.IOException: File copy failed: hdfs://nameservice1/filena --> s3a://cof-prod-lake-card/src/seam/acct_scores/acctmdlscore_card_cobna_anon_vldtd/instnc_id=20161023000000/000004_0_copy_6\r\n at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)\r\n at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)\r\n at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\r\n at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)\r\n at java.security.AccessController.doPrivileged(Native Method)\r\n at javax.security.auth.Subject.doAs(Subject.java:422)\r\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)\r\n at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\r\nCaused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://nameservice1/filename to s3a://bucketname/filename\r\n at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280)\r\n ... 10 more\r\nCaused by: com.cloudera.com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:164)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:299)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:77)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:74)\r\n at com.cloudera.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\r\n at com.cloudera.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:653)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1096)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:1279)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:1268)\r\n at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:257)\r\n at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)\r\n at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)\r\n at java.io.FilterOutputStream.close(FilterOutputStream.java:159)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:261)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:184)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:124)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)\r\n at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)\r\n ... 11 more\r\nCaused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 2; XML document structures must start and end within the same entity.\r\n at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)\r\n at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityScanner.skipChar(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\r\n at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\r\n at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\r\n at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)\r\n at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:151)\r\n ... 35 more\r\n\r\nAnd also please help me in choosing the number of mappers and what should I do to copy the data faster to S3.", "query_30": "From: reporter\nSubject: Getting file not found exception while using distcp with s3a\n\nI'm using the distcp option to copy the huge files from Hadoop to S3. Sometimes i'm getting the below error,\r\n\r\n*Command:* (Copying 378 GB data)\r\n\r\n_hadoop distcp -D HADOOP_OPTS=-Xmx12g -D HADOOP_CLIENT_OPTS='-Xmx12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled' -D 'mapreduce.map.memory.mb=12288' -D 'mapreduce.map.java.opts=-Xmx10g' -D 'mapreduce.reduce.memory.mb=12288' -D 'mapreduce.reduce.java.opts=-Xmx10g' '-Dfs.s3a.proxy.host=edhmgrn-prod.cloud.capitalone.com' '-Dfs.s3a.proxy.port=8088' '-Dfs.s3a.access.key=XXXXXXX' '-Dfs.s3a.secret.key=XXXXXXX' '-Dfs.s3a.connection.timeout=180000' '-Dfs.s3a.attempts.maximum=5' '-Dfs.s3a.fast.upload=true' '-Dfs.s3a.fast.upload.buffer=array' '-Dfs.s3a.fast.upload.active.blocks=50' '-Dfs.s3a.multipart.size=262144000' '-Dfs.s3a.threads.max=500' '-Dfs.s3a.threads.keepalivetime=600' '-Dfs.s3a.server-side-encryption-algorithm=AES256' -bandwidth 3072 -strategy dynamic -m 220 -numListstatusThreads 30 /src/ s3a://bucket/dest\r\n_\r\n17/11/01 12:23:27 INFO mapreduce.Job: Task Id : attempt_1497120915913_2792335_m_000165_0, Status : FAILED\r\nError: java.io.FileNotFoundException: No such file or directory: s3a://bucketname/filename\r\n\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1132)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:78)\r\n at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:197)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:256)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)\r\n at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)\r\n at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\r\n at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)\r\n at java.security.AccessController.doPrivileged(Native Method)\r\n at javax.security.auth.Subject.doAs(Subject.java:422)\r\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)\r\n at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\r\n\r\n17/11/01 12:28:32 INFO mapreduce.Job: Task Id : attempt_1497120915913_2792335_m_000010_0, Status : FAILED\r\nError: java.io.IOException: File copy failed: hdfs://nameservice1/filena --> s3a://cof-prod-lake-card/src/seam/acct_scores/acctmdlscore_card_cobna_anon_vldtd/instnc_id=20161023000000/000004_0_copy_6\r\n at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)\r\n at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)\r\n at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\r\n at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)\r\n at java.security.AccessController.doPrivileged(Native Method)\r\n at javax.security.auth.Subject.doAs(Subject.java:422)\r\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)\r\n at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\r\nCaused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://nameservice1/filename to s3a://bucketname/filename\r\n at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280)\r\n ... 10 more\r\nCaused by: com.cloudera.com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:164)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:299)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:77)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:74)\r\n at com.cloudera.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\r\n at com.cloudera.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:653)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1096)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:1279)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:1268)\r\n at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:257)\r\n at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)\r\n at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)\r\n at java.io.FilterOutputStream.close(FilterOutputStream.java:159)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:261)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:184)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:124)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)\r\n at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)\r\n ... 11 more\r\nCaused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 2; XML document structures must start and end within the same entity.\r\n at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)\r\n at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityScanner.skipChar(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\r\n at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\r\n at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\r\n at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)\r\n at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:151)\r\n ... 35 more\r\n\r\nAnd also please help me in choosing the number of mappers and what should I do to copy the data faster to S3.\n\n---\n\nFrom: developer\n\nLooks like you are hit by S3's eventual consistency.\r\n\r\nCheck out S3Guard which should help with your problem:\r\nhttps://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/\r\nhttps://hortonworks.com/blog/s3guard-amazon-s3-consistency/", "query_60": "From: reporter\nSubject: Getting file not found exception while using distcp with s3a\n\nI'm using the distcp option to copy the huge files from Hadoop to S3. Sometimes i'm getting the below error,\r\n\r\n*Command:* (Copying 378 GB data)\r\n\r\n_hadoop distcp -D HADOOP_OPTS=-Xmx12g -D HADOOP_CLIENT_OPTS='-Xmx12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled' -D 'mapreduce.map.memory.mb=12288' -D 'mapreduce.map.java.opts=-Xmx10g' -D 'mapreduce.reduce.memory.mb=12288' -D 'mapreduce.reduce.java.opts=-Xmx10g' '-Dfs.s3a.proxy.host=edhmgrn-prod.cloud.capitalone.com' '-Dfs.s3a.proxy.port=8088' '-Dfs.s3a.access.key=XXXXXXX' '-Dfs.s3a.secret.key=XXXXXXX' '-Dfs.s3a.connection.timeout=180000' '-Dfs.s3a.attempts.maximum=5' '-Dfs.s3a.fast.upload=true' '-Dfs.s3a.fast.upload.buffer=array' '-Dfs.s3a.fast.upload.active.blocks=50' '-Dfs.s3a.multipart.size=262144000' '-Dfs.s3a.threads.max=500' '-Dfs.s3a.threads.keepalivetime=600' '-Dfs.s3a.server-side-encryption-algorithm=AES256' -bandwidth 3072 -strategy dynamic -m 220 -numListstatusThreads 30 /src/ s3a://bucket/dest\r\n_\r\n17/11/01 12:23:27 INFO mapreduce.Job: Task Id : attempt_1497120915913_2792335_m_000165_0, Status : FAILED\r\nError: java.io.FileNotFoundException: No such file or directory: s3a://bucketname/filename\r\n\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1132)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:78)\r\n at org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:197)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:256)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)\r\n at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)\r\n at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\r\n at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)\r\n at java.security.AccessController.doPrivileged(Native Method)\r\n at javax.security.auth.Subject.doAs(Subject.java:422)\r\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)\r\n at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\r\n\r\n17/11/01 12:28:32 INFO mapreduce.Job: Task Id : attempt_1497120915913_2792335_m_000010_0, Status : FAILED\r\nError: java.io.IOException: File copy failed: hdfs://nameservice1/filena --> s3a://cof-prod-lake-card/src/seam/acct_scores/acctmdlscore_card_cobna_anon_vldtd/instnc_id=20161023000000/000004_0_copy_6\r\n at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)\r\n at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)\r\n at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)\r\n at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)\r\n at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)\r\n at java.security.AccessController.doPrivileged(Native Method)\r\n at javax.security.auth.Subject.doAs(Subject.java:422)\r\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)\r\n at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)\r\nCaused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://nameservice1/filename to s3a://bucketname/filename\r\n at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)\r\n at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280)\r\n ... 10 more\r\nCaused by: com.cloudera.com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:164)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:299)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:77)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:74)\r\n at com.cloudera.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\r\n at com.cloudera.com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)\r\n at com.cloudera.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738)\r\n at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:653)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1096)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:1279)\r\n at org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:1268)\r\n at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:257)\r\n at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)\r\n at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)\r\n at java.io.FilterOutputStream.close(FilterOutputStream.java:159)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:261)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:184)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:124)\r\n at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100)\r\n at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)\r\n ... 11 more\r\nCaused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 2; XML document structures must start and end within the same entity.\r\n at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)\r\n at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)\r\n at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)\r\n at org.apache.xerces.impl.XMLEntityScanner.skipChar(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source)\r\n at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\r\n at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\r\n at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\r\n at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)\r\n at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)\r\n at com.cloudera.com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:151)\r\n ... 35 more\r\n\r\nAnd also please help me in choosing the number of mappers and what should I do to copy the data faster to S3.\n\n---\n\nFrom: developer\n\nLooks like you are hit by S3's eventual consistency.\r\n\r\nCheck out S3Guard which should help with your problem:\r\nhttps://blog.cloudera.com/blog/2017/08/introducing-s3guard-s3-consistency-for-apache-hadoop/\r\nhttps://hortonworks.com/blog/s3guard-amazon-s3-consistency/\n\n---\n\nFrom: developer\n\nBut our production environment doesn't offer a Dynamo DB instance for S3 Guard. Is there a way to tune the options for distcp to copy the huge files. I'm looking for below information,\r\n\r\n1) How to select the number of map and it's size. I have a directory which has ~10000+ files with total size of ~250 GB. When I run with below option, it is taking ~1.30 hours.\r\n\r\nhadoop distcp -D HADOOP_OPTS=-Xmx12g -D HADOOP_CLIENT_OPTS='-Xmx12g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled' -D 'mapreduce.map.memory.mb=12288' -D 'mapreduce.map.java.opts=-Xmx10g' -D 'mapreduce.reduce.memory.mb=12288' -D 'mapreduce.reduce.java.opts=-Xmx10g' '-Dfs.s3a.proxy.host=edhmgrn-prod.cloud.capitalone.com' '-Dfs.s3a.proxy.port=8088' '-Dfs.s3a.access.key=XXXXXXX' '-Dfs.s3a.secret.key=XXXXXXX' '-Dfs.s3a.connection.timeout=180000' '-Dfs.s3a.attempts.maximum=5' '-Dfs.s3a.fast.upload=true' '-Dfs.s3a.fast.upload.buffer=array' '-Dfs.s3a.fast.upload.active.blocks=50' '-Dfs.s3a.multipart.size=262144000' '-Dfs.s3a.threads.max=500' '-Dfs.s3a.threads.keepalivetime=600' '-Dfs.s3a.server-side-encryption-algorithm=AES256' -bandwidth 3072 -strategy dynamic -m 200 -numListstatusThreads 30 /src/ s3a://bucket/dest\r\n\r\n2) I'm not seeing the throughput of 3gbps even after configuring the -bandwidth as 3072. \r\n\r\n3) How to configure the Java heap and map size for the huge file, so that distcp will give better performance.\r\n\r\n4) WIth fast upload option, I'm writing the files to S3 using threads. Could you please help me in providing some tuning option for this.\r\n\r\nAppreciate Your Help.", "target_key": "HADOOP-13145"} {"cluster": "JIRA-HADOOP-d3dfc3a59a1f", "issue_id": "13358149", "key": "HADOOP-17522", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "HADOOP", "query_0": "From: reporter\nSubject: Looking to build Hadoop but start-build-env.sh errors\n\nHey everyone, we're trying to build Hadoop from source and in building.txt we see \r\n\r\n \r\n\r\nThe easiest way to get an environment with all the appropriate tools is by means\r\nof the provided Docker config.\r\nThis requires a recent version of docker (1.4.1 and higher are known to work).\r\n\r\nOn Linux / Mac:\r\n Install Docker and run this command:\r\n\r\n$ ./start-build-env.sh\r\n\r\n \r\n\r\nHowever, and admittedly this is on 3.1.4, which I want to build - \r\n\r\n \r\n\r\n \r\n\r\n{{Step 26/36 : RUN pip2 install     configparser==4.0.2     pylint==1.9.2}}{{ ---> Running in acf678bc8314}}{{Collecting configparser==4.0.2}}{{  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl}}{{Collecting pylint==1.9.2}}{{  Downloading https://files.pythonhosted.org/packages/f2/95/0ca03c818ba3cd14f2dd4e95df5b7fa232424b7fc6ea1748d27f293bc007/pylint-1.9.2-py2.py3-none-any.whl (690kB)}}{{Collecting singledispatch; python_version < \"3.4\" (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl}}{{Collecting isort>=4.2.5 (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/a2/f7/f50fc9555dc0fe2dc1e7f69d93f71961d052857c296cad0fb6d275b20008/isort-5.7.0.tar.gz (169kB)}}{{Collecting astroid<2.0,>=1.6 (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/8b/29/0f7ec6fbf28a158886b7de49aee3a77a8a47a7e24c60e9fd6ec98ee2ec02/astroid-1.6.6-py2.py3-none-any.whl (305kB)}}{{Collecting backports.functools-lru-cache; python_version == \"2.7\" (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/da/d1/080d2bb13773803648281a49e3918f65b31b7beebf009887a529357fd44a/backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl}}{{Collecting mccabe (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl}}{{Collecting six (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl}}{{Collecting enum34>=1.1.3; python_version < \"3.4\" (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/6f/2c/a9386903ece2ea85e9807e0e062174dc26fdce8b05f216d00491be29fad5/enum34-1.1.10-py2-none-any.whl}}{{Collecting wrapt (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/82/f7/e43cefbe88c5fd371f4cf0cf5eb3feccd07515af9fd6cf7dbf1d1793a797/wrapt-1.12.1.tar.gz}}{{Collecting lazy-object-proxy (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/69/fc/79080e582c17b7f45ac0bb4a13b0260992dcc2519b5a29ac6cde3e81b6fa/lazy_object_proxy-1.5.2-cp27-cp27mu-manylinux1_x86_64.whl (52kB)}}{{Building wheels for collected packages: isort, wrapt}}{{  Running setup.py bdist_wheel for isort: started}}{{  Running setup.py bdist_wheel for isort: finished with status 'error'}}{{  Complete output from command /usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" bdist_wheel -d /tmp/tmp60t8Japip-wheel- --python-tag cp27:}}{{  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'}}{{    warnings.warn(msg)}}{{  running bdist_wheel}}{{  running build}}{{  running build_py}}{{  creating build}}{{  creating build/lib.linux-x86_64-2.7}}{{  creating build/lib.linux-x86_64-2.7/isort}}{{  copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/files.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/format.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/place.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/__main__.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/output.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/core.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/io.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/main.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/api.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/__init__.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort}}{{  creating build/lib.linux-x86_64-2.7/isort/_future}}{{  copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{  copying isort/_future/__init__.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{  creating build/lib.linux-x86_64-2.7/isort/_vendored}}{{  creating build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/__init__.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  creating build/lib.linux-x86_64-2.7/isort/deprecated}}{{  copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{  copying isort/deprecated/__init__.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{  creating build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/__init__.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  error: can't copy 'isort/stdlibs': doesn't exist or not a regular file}}{{  }}{{  ----------------------------------------}}{{  Failed building wheel for isort}}{{  Running setup.py clean for isort}}{{  Running setup.py bdist_wheel for wrapt: started}}{{  Running setup.py bdist_wheel for wrapt: finished with status 'done'}}{{  Stored in directory: /root/.cache/pip/wheels/b1/c2/ed/d62208260edbd3fa7156545c00ef966f45f2063d0a84f8208a}}{{Successfully built wrapt}}{{Failed to build isort}}{{Installing collected packages: configparser, six, singledispatch, isort, enum34, wrapt, backports.functools-lru-cache, lazy-object-proxy, astroid, mccabe, pylint}}{{  Running setup.py install for isort: started}}{{    Running setup.py install for isort: finished with status 'error'}}{{    Complete output from command /usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile:}}{{    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'}}{{      warnings.warn(msg)}}{{    running install}}{{    running build}}{{    running build_py}}{{    creating build}}{{    creating build/lib.linux-x86_64-2.7}}{{    creating build/lib.linux-x86_64-2.7/isort}}{{    copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/files.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/format.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/place.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/__main__.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/output.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/core.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/io.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/main.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/api.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/__init__.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort}}{{    creating build/lib.linux-x86_64-2.7/isort/_future}}{{    copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{    copying isort/_future/__init__.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{    creating build/lib.linux-x86_64-2.7/isort/_vendored}}{{    creating build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/__init__.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    creating build/lib.linux-x86_64-2.7/isort/deprecated}}{{    copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{    copying isort/deprecated/__init__.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{    creating build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/__init__.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    error: can't copy 'isort/stdlibs': doesn't exist or not a regular file}}{{    }}{{    ----------------------------------------}}{{Command \"/usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile\" failed with error code 1 in /tmp/pip-build-u9WX19/isort/}}\r\n\r\nYou are using pip version 8.1.1, however version 21.0.1 is available.\r\n\r\nYou should consider upgrading via the 'pip install --upgrade pip' command.\r\n\r\n \r\n\r\nIf I add that into the Dockerfile, I get\r\n\r\n{{}}{{ ---> Running in 5d4c46c61d29}}{{Traceback (most recent call last):}}{{  File \"/usr/local/bin/pip2\", line 7, in }}{{    from pip._internal.cli.main import main}}{{  File \"/usr/local/lib/python2.7/dist-packages/pip/_internal/cli/main.py\", line 60}}{{    sys.stderr.write(f\"ERROR: \\{exc}\")}}{{                                   ^}}{{SyntaxError: invalid syntax}}", "query_30": "From: reporter\nSubject: Looking to build Hadoop but start-build-env.sh errors\n\nHey everyone, we're trying to build Hadoop from source and in building.txt we see \r\n\r\n \r\n\r\nThe easiest way to get an environment with all the appropriate tools is by means\r\nof the provided Docker config.\r\nThis requires a recent version of docker (1.4.1 and higher are known to work).\r\n\r\nOn Linux / Mac:\r\n Install Docker and run this command:\r\n\r\n$ ./start-build-env.sh\r\n\r\n \r\n\r\nHowever, and admittedly this is on 3.1.4, which I want to build - \r\n\r\n \r\n\r\n \r\n\r\n{{Step 26/36 : RUN pip2 install     configparser==4.0.2     pylint==1.9.2}}{{ ---> Running in acf678bc8314}}{{Collecting configparser==4.0.2}}{{  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl}}{{Collecting pylint==1.9.2}}{{  Downloading https://files.pythonhosted.org/packages/f2/95/0ca03c818ba3cd14f2dd4e95df5b7fa232424b7fc6ea1748d27f293bc007/pylint-1.9.2-py2.py3-none-any.whl (690kB)}}{{Collecting singledispatch; python_version < \"3.4\" (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl}}{{Collecting isort>=4.2.5 (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/a2/f7/f50fc9555dc0fe2dc1e7f69d93f71961d052857c296cad0fb6d275b20008/isort-5.7.0.tar.gz (169kB)}}{{Collecting astroid<2.0,>=1.6 (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/8b/29/0f7ec6fbf28a158886b7de49aee3a77a8a47a7e24c60e9fd6ec98ee2ec02/astroid-1.6.6-py2.py3-none-any.whl (305kB)}}{{Collecting backports.functools-lru-cache; python_version == \"2.7\" (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/da/d1/080d2bb13773803648281a49e3918f65b31b7beebf009887a529357fd44a/backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl}}{{Collecting mccabe (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl}}{{Collecting six (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl}}{{Collecting enum34>=1.1.3; python_version < \"3.4\" (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/6f/2c/a9386903ece2ea85e9807e0e062174dc26fdce8b05f216d00491be29fad5/enum34-1.1.10-py2-none-any.whl}}{{Collecting wrapt (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/82/f7/e43cefbe88c5fd371f4cf0cf5eb3feccd07515af9fd6cf7dbf1d1793a797/wrapt-1.12.1.tar.gz}}{{Collecting lazy-object-proxy (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/69/fc/79080e582c17b7f45ac0bb4a13b0260992dcc2519b5a29ac6cde3e81b6fa/lazy_object_proxy-1.5.2-cp27-cp27mu-manylinux1_x86_64.whl (52kB)}}{{Building wheels for collected packages: isort, wrapt}}{{  Running setup.py bdist_wheel for isort: started}}{{  Running setup.py bdist_wheel for isort: finished with status 'error'}}{{  Complete output from command /usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" bdist_wheel -d /tmp/tmp60t8Japip-wheel- --python-tag cp27:}}{{  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'}}{{    warnings.warn(msg)}}{{  running bdist_wheel}}{{  running build}}{{  running build_py}}{{  creating build}}{{  creating build/lib.linux-x86_64-2.7}}{{  creating build/lib.linux-x86_64-2.7/isort}}{{  copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/files.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/format.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/place.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/__main__.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/output.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/core.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/io.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/main.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/api.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/__init__.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort}}{{  creating build/lib.linux-x86_64-2.7/isort/_future}}{{  copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{  copying isort/_future/__init__.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{  creating build/lib.linux-x86_64-2.7/isort/_vendored}}{{  creating build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/__init__.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  creating build/lib.linux-x86_64-2.7/isort/deprecated}}{{  copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{  copying isort/deprecated/__init__.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{  creating build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/__init__.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  error: can't copy 'isort/stdlibs': doesn't exist or not a regular file}}{{  }}{{  ----------------------------------------}}{{  Failed building wheel for isort}}{{  Running setup.py clean for isort}}{{  Running setup.py bdist_wheel for wrapt: started}}{{  Running setup.py bdist_wheel for wrapt: finished with status 'done'}}{{  Stored in directory: /root/.cache/pip/wheels/b1/c2/ed/d62208260edbd3fa7156545c00ef966f45f2063d0a84f8208a}}{{Successfully built wrapt}}{{Failed to build isort}}{{Installing collected packages: configparser, six, singledispatch, isort, enum34, wrapt, backports.functools-lru-cache, lazy-object-proxy, astroid, mccabe, pylint}}{{  Running setup.py install for isort: started}}{{    Running setup.py install for isort: finished with status 'error'}}{{    Complete output from command /usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile:}}{{    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'}}{{      warnings.warn(msg)}}{{    running install}}{{    running build}}{{    running build_py}}{{    creating build}}{{    creating build/lib.linux-x86_64-2.7}}{{    creating build/lib.linux-x86_64-2.7/isort}}{{    copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/files.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/format.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/place.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/__main__.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/output.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/core.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/io.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/main.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/api.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/__init__.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort}}{{    creating build/lib.linux-x86_64-2.7/isort/_future}}{{    copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{    copying isort/_future/__init__.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{    creating build/lib.linux-x86_64-2.7/isort/_vendored}}{{    creating build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/__init__.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    creating build/lib.linux-x86_64-2.7/isort/deprecated}}{{    copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{    copying isort/deprecated/__init__.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{    creating build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/__init__.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    error: can't copy 'isort/stdlibs': doesn't exist or not a regular file}}{{    }}{{    ----------------------------------------}}{{Command \"/usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile\" failed with error code 1 in /tmp/pip-build-u9WX19/isort/}}\r\n\r\nYou are using pip version 8.1.1, however version 21.0.1 is available.\r\n\r\nYou should consider upgrading via the 'pip install --upgrade pip' command.\r\n\r\n \r\n\r\nIf I add that into the Dockerfile, I get\r\n\r\n{{}}{{ ---> Running in 5d4c46c61d29}}{{Traceback (most recent call last):}}{{  File \"/usr/local/bin/pip2\", line 7, in }}{{    from pip._internal.cli.main import main}}{{  File \"/usr/local/lib/python2.7/dist-packages/pip/_internal/cli/main.py\", line 60}}{{    sys.stderr.write(f\"ERROR: \\{exc}\")}}{{                                   ^}}{{SyntaxError: invalid syntax}}\n\n---\n\nFrom: developer\n\nAdding this in the Dockerfile works great and gets me past that step!\r\n \r\nRUN pip install --upgrade pip==9.0.0\r\n ", "query_60": "From: reporter\nSubject: Looking to build Hadoop but start-build-env.sh errors\n\nHey everyone, we're trying to build Hadoop from source and in building.txt we see \r\n\r\n \r\n\r\nThe easiest way to get an environment with all the appropriate tools is by means\r\nof the provided Docker config.\r\nThis requires a recent version of docker (1.4.1 and higher are known to work).\r\n\r\nOn Linux / Mac:\r\n Install Docker and run this command:\r\n\r\n$ ./start-build-env.sh\r\n\r\n \r\n\r\nHowever, and admittedly this is on 3.1.4, which I want to build - \r\n\r\n \r\n\r\n \r\n\r\n{{Step 26/36 : RUN pip2 install     configparser==4.0.2     pylint==1.9.2}}{{ ---> Running in acf678bc8314}}{{Collecting configparser==4.0.2}}{{  Downloading https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl}}{{Collecting pylint==1.9.2}}{{  Downloading https://files.pythonhosted.org/packages/f2/95/0ca03c818ba3cd14f2dd4e95df5b7fa232424b7fc6ea1748d27f293bc007/pylint-1.9.2-py2.py3-none-any.whl (690kB)}}{{Collecting singledispatch; python_version < \"3.4\" (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/c5/10/369f50bcd4621b263927b0a1519987a04383d4a98fb10438042ad410cf88/singledispatch-3.4.0.3-py2.py3-none-any.whl}}{{Collecting isort>=4.2.5 (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/a2/f7/f50fc9555dc0fe2dc1e7f69d93f71961d052857c296cad0fb6d275b20008/isort-5.7.0.tar.gz (169kB)}}{{Collecting astroid<2.0,>=1.6 (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/8b/29/0f7ec6fbf28a158886b7de49aee3a77a8a47a7e24c60e9fd6ec98ee2ec02/astroid-1.6.6-py2.py3-none-any.whl (305kB)}}{{Collecting backports.functools-lru-cache; python_version == \"2.7\" (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/da/d1/080d2bb13773803648281a49e3918f65b31b7beebf009887a529357fd44a/backports.functools_lru_cache-1.6.1-py2.py3-none-any.whl}}{{Collecting mccabe (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/87/89/479dc97e18549e21354893e4ee4ef36db1d237534982482c3681ee6e7b57/mccabe-0.6.1-py2.py3-none-any.whl}}{{Collecting six (from pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl}}{{Collecting enum34>=1.1.3; python_version < \"3.4\" (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/6f/2c/a9386903ece2ea85e9807e0e062174dc26fdce8b05f216d00491be29fad5/enum34-1.1.10-py2-none-any.whl}}{{Collecting wrapt (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/82/f7/e43cefbe88c5fd371f4cf0cf5eb3feccd07515af9fd6cf7dbf1d1793a797/wrapt-1.12.1.tar.gz}}{{Collecting lazy-object-proxy (from astroid<2.0,>=1.6->pylint==1.9.2)}}{{  Downloading https://files.pythonhosted.org/packages/69/fc/79080e582c17b7f45ac0bb4a13b0260992dcc2519b5a29ac6cde3e81b6fa/lazy_object_proxy-1.5.2-cp27-cp27mu-manylinux1_x86_64.whl (52kB)}}{{Building wheels for collected packages: isort, wrapt}}{{  Running setup.py bdist_wheel for isort: started}}{{  Running setup.py bdist_wheel for isort: finished with status 'error'}}{{  Complete output from command /usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" bdist_wheel -d /tmp/tmp60t8Japip-wheel- --python-tag cp27:}}{{  /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'}}{{    warnings.warn(msg)}}{{  running bdist_wheel}}{{  running build}}{{  running build_py}}{{  creating build}}{{  creating build/lib.linux-x86_64-2.7}}{{  creating build/lib.linux-x86_64-2.7/isort}}{{  copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/files.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/format.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/place.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/__main__.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/output.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/core.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/io.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/main.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/api.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/__init__.py -> build/lib.linux-x86_64-2.7/isort}}{{  copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort}}{{  creating build/lib.linux-x86_64-2.7/isort/_future}}{{  copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{  copying isort/_future/__init__.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{  creating build/lib.linux-x86_64-2.7/isort/_vendored}}{{  creating build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  copying isort/_vendored/toml/__init__.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{  creating build/lib.linux-x86_64-2.7/isort/deprecated}}{{  copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{  copying isort/deprecated/__init__.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{  creating build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  copying isort/stdlibs/__init__.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{  error: can't copy 'isort/stdlibs': doesn't exist or not a regular file}}{{  }}{{  ----------------------------------------}}{{  Failed building wheel for isort}}{{  Running setup.py clean for isort}}{{  Running setup.py bdist_wheel for wrapt: started}}{{  Running setup.py bdist_wheel for wrapt: finished with status 'done'}}{{  Stored in directory: /root/.cache/pip/wheels/b1/c2/ed/d62208260edbd3fa7156545c00ef966f45f2063d0a84f8208a}}{{Successfully built wrapt}}{{Failed to build isort}}{{Installing collected packages: configparser, six, singledispatch, isort, enum34, wrapt, backports.functools-lru-cache, lazy-object-proxy, astroid, mccabe, pylint}}{{  Running setup.py install for isort: started}}{{    Running setup.py install for isort: finished with status 'error'}}{{    Complete output from command /usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile:}}{{    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'}}{{      warnings.warn(msg)}}{{    running install}}{{    running build}}{{    running build_py}}{{    creating build}}{{    creating build/lib.linux-x86_64-2.7}}{{    creating build/lib.linux-x86_64-2.7/isort}}{{    copying isort/_version.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/settings.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/logo.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/hooks.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/wrap_modes.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/sections.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/parse.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/utils.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/setuptools_commands.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/files.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/format.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/wrap.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/place.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/profiles.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/__main__.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/output.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/core.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/exceptions.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/pylama_isort.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/literal.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/comments.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/io.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/identify.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/main.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/api.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/__init__.py -> build/lib.linux-x86_64-2.7/isort}}{{    copying isort/sorting.py -> build/lib.linux-x86_64-2.7/isort}}{{    creating build/lib.linux-x86_64-2.7/isort/_future}}{{    copying isort/_future/_dataclasses.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{    copying isort/_future/__init__.py -> build/lib.linux-x86_64-2.7/isort/_future}}{{    creating build/lib.linux-x86_64-2.7/isort/_vendored}}{{    creating build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/ordered.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/tz.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/decoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/encoder.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    copying isort/_vendored/toml/__init__.py -> build/lib.linux-x86_64-2.7/isort/_vendored/toml}}{{    creating build/lib.linux-x86_64-2.7/isort/deprecated}}{{    copying isort/deprecated/finders.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{    copying isort/deprecated/__init__.py -> build/lib.linux-x86_64-2.7/isort/deprecated}}{{    creating build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py38.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py39.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py35.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py37.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/all.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py3.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py27.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py2.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/py36.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    copying isort/stdlibs/__init__.py -> build/lib.linux-x86_64-2.7/isort/stdlibs}}{{    error: can't copy 'isort/stdlibs': doesn't exist or not a regular file}}{{    }}{{    ----------------------------------------}}{{Command \"/usr/bin/python -u -c \"import setuptools, tokenize;__file__='/tmp/pip-build-u9WX19/isort/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))\" install --record /tmp/pip-Pa5mnZ-record/install-record.txt --single-version-externally-managed --compile\" failed with error code 1 in /tmp/pip-build-u9WX19/isort/}}\r\n\r\nYou are using pip version 8.1.1, however version 21.0.1 is available.\r\n\r\nYou should consider upgrading via the 'pip install --upgrade pip' command.\r\n\r\n \r\n\r\nIf I add that into the Dockerfile, I get\r\n\r\n{{}}{{ ---> Running in 5d4c46c61d29}}{{Traceback (most recent call last):}}{{  File \"/usr/local/bin/pip2\", line 7, in }}{{    from pip._internal.cli.main import main}}{{  File \"/usr/local/lib/python2.7/dist-packages/pip/_internal/cli/main.py\", line 60}}{{    sys.stderr.write(f\"ERROR: \\{exc}\")}}{{                                   ^}}{{SyntaxError: invalid syntax}}\n\n---\n\nFrom: developer\n\nAdding this in the Dockerfile works great and gets me past that step!\r\n \r\nRUN pip install --upgrade pip==9.0.0\r\n \n\n---\n\nFrom: developer\n\nh1. {{Sad times}}\r\n\r\n \r\n\r\n{{WARN engine npm@7.5.3: wanted: \\{\"node\":\">=10\"} (current: \\{\"node\":\"4.2.6\",\"npm\":\"3.5.2\"})}}{{WARN engine npm@7.5.3: wanted: \\{\"node\":\">=10\"} (current: \\{\"node\":\"4.2.6\",\"npm\":\"3.5.2\"})}}{{/usr/local/lib}}{{`-- (empty)}}{{ }}{{npm ERR! Linux 4.19.121-linuxkit}}{{npm ERR! argv \"/usr/bin/nodejs\" \"/usr/bin/npm\" \"install\" \"npm@latest\" \"-g\"}}{{npm ERR! node v4.2.6}}{{npm ERR! npm  v3.5.2}}{{npm ERR! path /usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552}}{{npm ERR! code ENOENT}}{{npm ERR! errno -2}}{{npm ERR! syscall rename}}{{ }}{{npm ERR! enoent ENOENT: no such file or directory, rename '/usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552' -> '/usr/local/lib/node_modules/npm/node_modules/@npmcli/ci-detect'}}{{npm ERR! enoent ENOENT: no such file or directory, rename '/usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552' -> '/usr/local/lib/node_modules/npm/node_modules/@npmcli/ci-detect'}}{{npm ERR! enoent This is most likely not a problem with npm itself}}{{npm ERR! enoent and is related to npm not being able to find a file.}}{{npm ERR! enoent }}{{ }}{{npm ERR! Please include the following file with any support request:}}{{npm ERR!     /root/npm-debug.log}}{{npm ERR! code 1}}{{The command '/bin/sh -c apt-get -y install nodejs &&     ln -s /usr/bin/nodejs /usr/bin/node &&     apt-get -y install npm &&     npm install npm@latest -g &&     npm install -g bower &&     npm install -g ember-cli' returned a non-zero code: 1}}", "target_key": "HADOOP-17120"} {"cluster": "JIRA-HADOOP-0de7dde1714d", "issue_id": "13382937", "key": "HADOOP-17755", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "HADOOP", "query_0": "From: reporter\nSubject: EOF reached error reading ORC file on S3A\n\nHi I am trying to do some transformation using Spark 3.1.1-Hadoop 3.2 on K8s and using s3a\r\n\r\nI have around 700 GB of data to read and around 200 executors (5 vCore and 30G each).\r\n\r\nIts able to read most of the files in problematic stage (Scan orc => Filter => Project) but is failing with few files at the end with below error.  The size of the file mentioned in error is around 140 MB and all other files are of similar size.\r\n\r\nI am able to read and rewrite the specific file mentioned which suggest the file is not corrupted.\r\n\r\nLet me know if further information is required\r\n\r\n \r\n{code:java}\r\njava.io.IOException: Error reading file: s3a:///part-00001-5e22a873-82a5-4781-9eb9-473b483396bd.c000.zlib.orcjava.io.IOException: Error reading file: s3a:///part-00001-5e22a873-82a5-4781-9eb9-473b483396bd.c000.zlib.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1331) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:96) at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:177) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: java.io.EOFException: End of file reached before reading fully. at org.apache.hadoop.fs.s3a.S3AInputStream.readFully(S3AInputStream.java:702) at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111) at org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:566) at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:285) at org.apache.orc.impl.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:1237) at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1105) at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1256) at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1291) at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1327) ... 20 more\r\n{code}\r\n \r\n\r\n ", "query_30": "From: reporter\nSubject: EOF reached error reading ORC file on S3A\n\nHi I am trying to do some transformation using Spark 3.1.1-Hadoop 3.2 on K8s and using s3a\r\n\r\nI have around 700 GB of data to read and around 200 executors (5 vCore and 30G each).\r\n\r\nIts able to read most of the files in problematic stage (Scan orc => Filter => Project) but is failing with few files at the end with below error.  The size of the file mentioned in error is around 140 MB and all other files are of similar size.\r\n\r\nI am able to read and rewrite the specific file mentioned which suggest the file is not corrupted.\r\n\r\nLet me know if further information is required\r\n\r\n \r\n{code:java}\r\njava.io.IOException: Error reading file: s3a:///part-00001-5e22a873-82a5-4781-9eb9-473b483396bd.c000.zlib.orcjava.io.IOException: Error reading file: s3a:///part-00001-5e22a873-82a5-4781-9eb9-473b483396bd.c000.zlib.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1331) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:96) at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:177) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: java.io.EOFException: End of file reached before reading fully. at org.apache.hadoop.fs.s3a.S3AInputStream.readFully(S3AInputStream.java:702) at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111) at org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:566) at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:285) at org.apache.orc.impl.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:1237) at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1105) at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1256) at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1291) at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1327) ... 20 more\r\n{code}\r\n \r\n\r\n \n\n---\n\nFrom: developer\n\nIs it always the same files? Or is this intermittent?\r\n\r\n1. Can you grab cloudstore and run the storediag command against the store & attach the output here. This does not print secrets, but you can still do a search and replace of bucket name etc if you want to be maximum rigorous, and cut those obfuscated secrets out. This will save me asking you lots of questions about the store/your config\r\n\r\n{code}\r\nhadoop jar cloudstore-1.0.jar storedciag -r s3a:///\r\n{code}\r\n\r\n\r\n2. test with later hadoop binaries, especially hadoop-3.3.0, or ideally even, the latest hadoop-3.3.1 RC?\r\n\r\n3. turn on debug logging on the stream: org.apache.hadoop.fs.s3a.S3AInputStream\r\n\r\n4. Does everything work if the file is not in S3? e.g. HDFS, file://, etc?\r\n\r\n\r\n", "query_60": "From: reporter\nSubject: EOF reached error reading ORC file on S3A\n\nHi I am trying to do some transformation using Spark 3.1.1-Hadoop 3.2 on K8s and using s3a\r\n\r\nI have around 700 GB of data to read and around 200 executors (5 vCore and 30G each).\r\n\r\nIts able to read most of the files in problematic stage (Scan orc => Filter => Project) but is failing with few files at the end with below error.  The size of the file mentioned in error is around 140 MB and all other files are of similar size.\r\n\r\nI am able to read and rewrite the specific file mentioned which suggest the file is not corrupted.\r\n\r\nLet me know if further information is required\r\n\r\n \r\n{code:java}\r\njava.io.IOException: Error reading file: s3a:///part-00001-5e22a873-82a5-4781-9eb9-473b483396bd.c000.zlib.orcjava.io.IOException: Error reading file: s3a:///part-00001-5e22a873-82a5-4781-9eb9-473b483396bd.c000.zlib.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1331) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:96) at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458) at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:177) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)Caused by: java.io.EOFException: End of file reached before reading fully. at org.apache.hadoop.fs.s3a.S3AInputStream.readFully(S3AInputStream.java:702) at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111) at org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:566) at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:285) at org.apache.orc.impl.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:1237) at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1105) at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1256) at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1291) at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1327) ... 20 more\r\n{code}\r\n \r\n\r\n \n\n---\n\nFrom: developer\n\nIs it always the same files? Or is this intermittent?\r\n\r\n1. Can you grab cloudstore and run the storediag command against the store & attach the output here. This does not print secrets, but you can still do a search and replace of bucket name etc if you want to be maximum rigorous, and cut those obfuscated secrets out. This will save me asking you lots of questions about the store/your config\r\n\r\n{code}\r\nhadoop jar cloudstore-1.0.jar storedciag -r s3a:///\r\n{code}\r\n\r\n\r\n2. test with later hadoop binaries, especially hadoop-3.3.0, or ideally even, the latest hadoop-3.3.1 RC?\r\n\r\n3. turn on debug logging on the stream: org.apache.hadoop.fs.s3a.S3AInputStream\r\n\r\n4. Does everything work if the file is not in S3? e.g. HDFS, file://, etc?\r\n\r\n\r\n\n\n---\n\nFrom: developer\n\nStill awaiting the requested information. If you can't supply it we will have to close as CANNOT-REPRODUCE.", "target_key": "HADOOP-16109"} {"cluster": "JIRA-HBASE-8cbaa6d5ed91", "issue_id": "13069062", "key": "HBASE-17992", "n_comments": 11, "progress_valid": {"0": true, "30": true, "60": true}, "project": "HBASE", "query_0": "From: reporter\nSubject: The snapShot TimeoutException causes the cleanerChore thread to fail to complete the archive correctly\n\nThe problem is that when the snapshot occurs TimeoutException or other Exceptions, there is no correct delete /hbase/.hbase-snapshot/tmp, which causes the cleanerChore to fail to complete the archive correctly.\nModifying the configuration parameter (hbase.snapshot.master.timeout.millis = 600000) only reduces the probability of the problem occurring.\n\nSo the solution to the problem is: multi-Threaded exceptions or TimeoutExceptions, the Main-thread must wait until all the tasks are finished or canceled, the Main-thread can be cleared /hbase/.hbase-snapshot/tmp/snapshotName.Otherwise the task is likely to write /hbase/.hbase-snapshot/tmp/snapshotName/region - mainfest\n\nThe problem exists in disabledTableSnapshot and enabledTableSnapshot, because I'm currently using the disabledTableSnapshot, so I provide the patch of disabledTableSnapshot", "query_30": "From: reporter\nSubject: The snapShot TimeoutException causes the cleanerChore thread to fail to complete the archive correctly\n\nThe problem is that when the snapshot occurs TimeoutException or other Exceptions, there is no correct delete /hbase/.hbase-snapshot/tmp, which causes the cleanerChore to fail to complete the archive correctly.\nModifying the configuration parameter (hbase.snapshot.master.timeout.millis = 600000) only reduces the probability of the problem occurring.\n\nSo the solution to the problem is: multi-Threaded exceptions or TimeoutExceptions, the Main-thread must wait until all the tasks are finished or canceled, the Main-thread can be cleared /hbase/.hbase-snapshot/tmp/snapshotName.Otherwise the task is likely to write /hbase/.hbase-snapshot/tmp/snapshotName/region - mainfest\n\nThe problem exists in disabledTableSnapshot and enabledTableSnapshot, because I'm currently using the disabledTableSnapshot, so I provide the patch of disabledTableSnapshot\n\n---\n\nFrom: developer\n\nCan you cover enabledTableSnapshot as well ?\n\n---\n\nFrom: developer\n\nokay\n\n---\n\nFrom: developer\n\nCan you put the patch on review board ?\n\nIs it possible to add a test for the scenario ?\n\nThanks\n\n---\n\nFrom: developer\n\n{code}\n+ while(!exec.isTerminated()){\n+ Thread.sleep(2000);\n{code}\nShould there be a bound on the total duration of waiting ?\n{code}\n+ this.waittimeAttempts =\n+ this.coord.getRpcs().getConfiguration()\n+ .getLong(\"hbase.procedure.clearznodes.waittime\", 3000l);\n{code}\nAdd dot between \"wait\" and \"time\" in the config key name. If you keep Attempts in variable name, you can use waitTimeBetweenAttempts.\n{code}\n+ private final Map> submitSubprocedures=\n+ new HashMap>();\n{code}\nAdd comment for what purpose the map serves.", "query_60": "From: reporter\nSubject: The snapShot TimeoutException causes the cleanerChore thread to fail to complete the archive correctly\n\nThe problem is that when the snapshot occurs TimeoutException or other Exceptions, there is no correct delete /hbase/.hbase-snapshot/tmp, which causes the cleanerChore to fail to complete the archive correctly.\nModifying the configuration parameter (hbase.snapshot.master.timeout.millis = 600000) only reduces the probability of the problem occurring.\n\nSo the solution to the problem is: multi-Threaded exceptions or TimeoutExceptions, the Main-thread must wait until all the tasks are finished or canceled, the Main-thread can be cleared /hbase/.hbase-snapshot/tmp/snapshotName.Otherwise the task is likely to write /hbase/.hbase-snapshot/tmp/snapshotName/region - mainfest\n\nThe problem exists in disabledTableSnapshot and enabledTableSnapshot, because I'm currently using the disabledTableSnapshot, so I provide the patch of disabledTableSnapshot\n\n---\n\nFrom: developer\n\nCan you cover enabledTableSnapshot as well ?\n\n---\n\nFrom: developer\n\nokay\n\n---\n\nFrom: developer\n\nCan you put the patch on review board ?\n\nIs it possible to add a test for the scenario ?\n\nThanks\n\n---\n\nFrom: developer\n\n{code}\n+ while(!exec.isTerminated()){\n+ Thread.sleep(2000);\n{code}\nShould there be a bound on the total duration of waiting ?\n{code}\n+ this.waittimeAttempts =\n+ this.coord.getRpcs().getConfiguration()\n+ .getLong(\"hbase.procedure.clearznodes.waittime\", 3000l);\n{code}\nAdd dot between \"wait\" and \"time\" in the config key name. If you keep Attempts in variable name, you can use waitTimeBetweenAttempts.\n{code}\n+ private final Map> submitSubprocedures=\n+ new HashMap>();\n{code}\nAdd comment for what purpose the map serves.\n\n---\n\nFrom: developer\n\nwhether the disabledTableSnapshot#exec needs to set the total duration of waiting?\n{code:title=DisabledTableSnapshotHandler.java|borderStyle=solid}\npublic void snapshotRegions(List> regionsAndLocations)\n throws IOException, KeeperException {\n ...\n ThreadPoolExecutor exec = SnapshotManifest.createExecutor(conf, \"DisabledTableSnapshot\");\n try {\n ModifyRegionUtils.editRegions(exec, regions, new ModifyRegionUtils.RegionEditTask() {\n @Override\n public void editRegion(final HRegionInfo regionInfo) throws IOException {\n snapshotManifest.addRegion(FSUtils.getTableDir(rootDir, snapshotTable), regionInfo);\n }\n });\n\t}catch(IOException e){\n exec.shutdownNow();\n while(!exec.isTerminated()){\n Thread.sleep(2000);\n }\n throw e;\n }\n exec.shutdown();\n ...\n}\n{code}\n\nSnapshotmanifest#addregion() : read memory and write HDFS\nRead memory -- does not take a long time\nWrite HDFS -- HDFS has its own timeout or exception handling\nAnd exec defaults to eight threads, and if an exception occurs, only >= 8 threads execute.\nSo I think there's no need set the total duration of waiting, for ensure that all task ends.\n\n\n---\n\nFrom: developer\n\n@Ted Yu\n{quote}Can you put the patch on review board ?{quote}\nReview board? Sorry, not familiar with:(\n\n{quote}Is it possible to add a test for the scenario ?{quote}\nI use eclipse(remote java application) to reexisting the problem, complete code modification and test pass.\nBut I don't know how to add a test for the scenario, sorry:(\n\n---\n\nFrom: developer\n\nYou can use reviews.apache.org to create review request.\nAdd hbase as the group so that we all receive the notification.", "target_key": "HBASE-16464"} {"cluster": "JIRA-HBASE-97916ddf28a1", "issue_id": "13243811", "key": "HBASE-22667", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "HBASE", "query_0": "From: reporter\nSubject: [Flush] NPE when region flushs\n\n{code}\r\n2019-07-09 08:02:14,262 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hostname,16020,1562233574704: Replay of WAL required. Forcing server shutdown\r\norg.apache.hadoop.hbase.DroppedSnapshotException: region: namespace:table,963,1562296120996.b8e2f19748d374d192b93f106a0f73b3.\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2646)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2322)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2284)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2170)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2095)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)\r\n at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.NullPointerException\r\n at java.util.ArrayList.(ArrayList.java:177)\r\n at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:863)\r\n at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1172)\r\n at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1145)\r\n at org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:122)\r\n at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2505)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2600)\r\n ... 9 more\r\n{code}", "query_30": "From: reporter\nSubject: [Flush] NPE when region flushs\n\n{code}\r\n2019-07-09 08:02:14,262 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hostname,16020,1562233574704: Replay of WAL required. Forcing server shutdown\r\norg.apache.hadoop.hbase.DroppedSnapshotException: region: namespace:table,963,1562296120996.b8e2f19748d374d192b93f106a0f73b3.\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2646)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2322)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2284)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2170)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2095)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)\r\n at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.NullPointerException\r\n at java.util.ArrayList.(ArrayList.java:177)\r\n at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:863)\r\n at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1172)\r\n at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1145)\r\n at org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:122)\r\n at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2505)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2600)\r\n ... 9 more\r\n{code}\n\n---\n\nFrom: developer\n\nI haven't dig out the root cause yet, just file it first.", "query_60": "From: reporter\nSubject: [Flush] NPE when region flushs\n\n{code}\r\n2019-07-09 08:02:14,262 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hostname,16020,1562233574704: Replay of WAL required. Forcing server shutdown\r\norg.apache.hadoop.hbase.DroppedSnapshotException: region: namespace:table,963,1562296120996.b8e2f19748d374d192b93f106a0f73b3.\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2646)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2322)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2284)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2170)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2095)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)\r\n at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)\r\n at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.NullPointerException\r\n at java.util.ArrayList.(ArrayList.java:177)\r\n at org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:863)\r\n at org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1172)\r\n at org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1145)\r\n at org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:122)\r\n at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2505)\r\n at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2600)\r\n ... 9 more\r\n{code}\n\n---\n\nFrom: developer\n\nI haven't dig out the root cause yet, just file it first.\n\n---\n\nFrom: developer\n\nIt's werid, since null check for memStoreScanners is done in line 855.\r\n{code}\r\n 852 @Override\r\n 853 public void updateReaders(List sfs, List memStoreScanners) throws IOException {\r\n 854 if (CollectionUtils.isEmpty(sfs)\r\n 855 && CollectionUtils.isEmpty(memStoreScanners)) {\r\n 856 return;\r\n 857 }\r\n 858 flushLock.lock();\r\n 859 try {\r\n 860 if (this.closing) {\r\n 861 // Lets close scanners created by caller, since close() won't notice this.\r\n 862 // memStoreScanners is immutable, so lets create a new list.\r\n 863 clearAndClose(new ArrayList<>(memStoreScanners));\r\n 864 return;\r\n 865 }\r\n{code}", "target_key": "HBASE-21069"} {"cluster": "JIRA-SPARK-34e8f81d6695", "issue_id": "12860596", "key": "SPARK-10358", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "SPARK", "query_0": "From: reporter\nSubject: Spark-sql throws IOException on exit when using HDFS to store event log.\n\nh2. Summary \nIn Spark 1.3.1, if using HDFS to store event log, spark-sql will throw an \"java.io.IOException: Filesystem closed\" when exit. \nh2. How to reproduce \n1. Enable event log mechanism, and configure the file location to HDFS. \n You can do this by setting these two properties in spark-defaults.conf: \nspark.eventLog.enabled true \nspark.eventLog.dir hdfs://xxxxx:xxxxx/spark-events \n2. start spark-sql, and type exit once it starts. \n{noformat} \nspark-sql> exit; \n15/08/14 06:29:20 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception \nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \nat java.lang.reflect.Method.invoke(Method.java:597) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat scala.Option.foreach(Option.scala:236) \nat org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:181) \nat org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:54) \nat org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) \nat org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) \nat org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53) \nat org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) \nat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1678) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60) \nCaused by: java.io.IOException: Filesystem closed \nat org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) \nat org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1985) \nat org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946) \nat org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) \n... 19 more \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} \n15/08/14 06:29:20 INFO ui.SparkUI: Stopped Spark web UI at http://9.111.251.177:4040 \n15/08/14 06:29:20 INFO scheduler.DAGScheduler: Stopping DAGScheduler \n15/08/14 06:29:20 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors \n15/08/14 06:29:20 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down \nException in thread \"Thread-4\" java.io.IOException: Filesystem closed \nat org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) \nat org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1986) \nat org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) \nat org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) \nat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) \nat org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) \nat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) \nat org.apache.spark.scheduler.EventLoggingListener.stop(EventLoggingListener.scala:198) \nat org.apache.spark.SparkContext$$anonfun$stop$4.apply(SparkContext.scala:1391) \nat org.apache.spark.SparkContext$$anonfun$stop$4.apply(SparkContext.scala:1391) \nat scala.Option.foreach(Option.scala:236) \nat org.apache.spark.SparkContext.stop(SparkContext.scala:1391) \nat org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.stop(SparkSQLEnv.scala:66) \nat org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$$anon$1.run(SparkSQLCLIDriver.scala:107) \n{noformat} \nh2. Analysis \nEnabling DEBUG level, we can see more information: \n{noformat} \n15/08/14 09:32:34 DEBUG SessionState: Removing resource dir /tmp/a9d3b5b8-3329-4e93-aee2-0a3496d661fa_resources \n15/08/14 09:32:34 DEBUG SparkSQLEnv: Shutting down Spark SQL Environment \n15/08/14 09:32:34 DEBUG DiskBlockManager: Shutdown hook called \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient writeChunk allocating new packet seqno=2, src=/spark/events/app-20150814093225-0002.snappy.inprogress, packetSize=65532, chunksPerPacket=127,bytesCurBlock=2048 \n15/08/14 09:32:34 DEBUG DFSClient: Queued packet 2 \n15/08/14 09:32:34 DEBUG DFSClient: Queued packet 3 \n15/08/14 09:32:34 DEBUG DFSClient: Waiting for ack for: 3 \n15/08/14 09:32:34 DEBUG Utils: Shutdown hook called \n15/08/14 09:32:34 DEBUG DFSClient: DataStreamer block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 sending packet packet seqno:2 offsetInBlock:2048 lastPacketInBlock:false lastByteOffsetInBlock: 2399 \n15/08/14 09:32:34 INFO SparkContext: END POST APPLICATION END \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient seqno: 2 status: SUCCESS downstreamAckTimeNanos: 0 \n15/08/14 09:32:34 DEBUG DFSClient: DataStreamer block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 sending packet packet seqno:3 offsetInBlock:2399 lastPacketInBlock:true lastByteOffsetInBlock: 2399 \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient seqno: 3 status: SUCCESS downstreamAckTimeNanos: 0 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping org.spark-project.jetty.server.Server@489bb457 \n15/08/14 09:32:34 DEBUG DFSClient: Closing old block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping SelectChannelConnector@0.0.0.0:4040 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping org.spark-project.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@15e3d24a \n15/08/14 09:32:34 DEBUG nio: Stopped Thread[qtp1717362942-45 Selector0,5,main] on org.spark-project.jetty.io.nio.SelectorManager$1@25964fe8 \n15/08/14 09:32:34 DEBUG Client: IPC Client (123257783) connection to /9.111.251.177:9000 from root sending #6 \n15/08/14 09:32:34 DEBUG Client: IPC Client (123257783) connection to /9.111.251.177:9000 from root got value #6 \n15/08/14 09:32:34 DEBUG ProtobufRpcEngine: Call: complete took 4ms \n15/08/14 09:32:34 DEBUG nio: Stopped Thread[qtp1717362942-46 Selector1,5,main] on org.spark-project.jetty.io.nio.SelectorManager$1@2f581b9f \n15/08/14 09:32:34 DEBUG Client: stopping client from cache: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: removing client from cache: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: Stopping client \n15/08/14 09:32:34 *ERROR* LiveListenerBus: Listener EventLoggingListener threw an exception \njava.lang.reflect.InvocationTargetException \nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \nat java.lang.reflect.Method.invoke(Method.java:597) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat scala.Option.foreach(Option.scala:236) \n{noformat} \n\nFrom the stacktrace, we can see a \"Shutdown hook called\" of org.apache.spark.util.Utils, and then the shutdown hook of HDFS where DFSClient is stopped, and finally the SparkSQLEnv.stop, where the exception was thrown. \nClearly, *Spark is trying to add new event log message after HDFS client is closed, which caused the exception*. \n\nAfter checking org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver, we found that the SparkSQLEnv.stop() is invoked from a normal java runtime shutdown hook: \n{code} \n // Clean up after we exit \n Runtime.getRuntime.addShutdownHook( \n new Thread() { \n override def run() { \n SparkSQLEnv.stop() \n } \n } \n ) \n{code} \nThis shutdown hook has lower priority, and will be executed after the DFSClient is closed. \nh2. Solution \nWe changed it by *using Hadoop's ShutdownHookManager and assigning a higher priority than HDFS's shutdown hook*. It fixed the problem. \nWe'll link a pull request later for review. \nThe link is [https://github.com/apache/spark/pull/8530#issuecomment-136236733]", "query_30": "From: reporter\nSubject: Spark-sql throws IOException on exit when using HDFS to store event log.\n\nh2. Summary \nIn Spark 1.3.1, if using HDFS to store event log, spark-sql will throw an \"java.io.IOException: Filesystem closed\" when exit. \nh2. How to reproduce \n1. Enable event log mechanism, and configure the file location to HDFS. \n You can do this by setting these two properties in spark-defaults.conf: \nspark.eventLog.enabled true \nspark.eventLog.dir hdfs://xxxxx:xxxxx/spark-events \n2. start spark-sql, and type exit once it starts. \n{noformat} \nspark-sql> exit; \n15/08/14 06:29:20 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception \nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \nat java.lang.reflect.Method.invoke(Method.java:597) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat scala.Option.foreach(Option.scala:236) \nat org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:181) \nat org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:54) \nat org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) \nat org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) \nat org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53) \nat org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) \nat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1678) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60) \nCaused by: java.io.IOException: Filesystem closed \nat org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) \nat org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1985) \nat org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946) \nat org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) \n... 19 more \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} \n15/08/14 06:29:20 INFO ui.SparkUI: Stopped Spark web UI at http://9.111.251.177:4040 \n15/08/14 06:29:20 INFO scheduler.DAGScheduler: Stopping DAGScheduler \n15/08/14 06:29:20 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors \n15/08/14 06:29:20 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down \nException in thread \"Thread-4\" java.io.IOException: Filesystem closed \nat org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) \nat org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1986) \nat org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) \nat org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) \nat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) \nat org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) \nat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) \nat org.apache.spark.scheduler.EventLoggingListener.stop(EventLoggingListener.scala:198) \nat org.apache.spark.SparkContext$$anonfun$stop$4.apply(SparkContext.scala:1391) \nat org.apache.spark.SparkContext$$anonfun$stop$4.apply(SparkContext.scala:1391) \nat scala.Option.foreach(Option.scala:236) \nat org.apache.spark.SparkContext.stop(SparkContext.scala:1391) \nat org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.stop(SparkSQLEnv.scala:66) \nat org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$$anon$1.run(SparkSQLCLIDriver.scala:107) \n{noformat} \nh2. Analysis \nEnabling DEBUG level, we can see more information: \n{noformat} \n15/08/14 09:32:34 DEBUG SessionState: Removing resource dir /tmp/a9d3b5b8-3329-4e93-aee2-0a3496d661fa_resources \n15/08/14 09:32:34 DEBUG SparkSQLEnv: Shutting down Spark SQL Environment \n15/08/14 09:32:34 DEBUG DiskBlockManager: Shutdown hook called \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient writeChunk allocating new packet seqno=2, src=/spark/events/app-20150814093225-0002.snappy.inprogress, packetSize=65532, chunksPerPacket=127,bytesCurBlock=2048 \n15/08/14 09:32:34 DEBUG DFSClient: Queued packet 2 \n15/08/14 09:32:34 DEBUG DFSClient: Queued packet 3 \n15/08/14 09:32:34 DEBUG DFSClient: Waiting for ack for: 3 \n15/08/14 09:32:34 DEBUG Utils: Shutdown hook called \n15/08/14 09:32:34 DEBUG DFSClient: DataStreamer block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 sending packet packet seqno:2 offsetInBlock:2048 lastPacketInBlock:false lastByteOffsetInBlock: 2399 \n15/08/14 09:32:34 INFO SparkContext: END POST APPLICATION END \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient seqno: 2 status: SUCCESS downstreamAckTimeNanos: 0 \n15/08/14 09:32:34 DEBUG DFSClient: DataStreamer block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 sending packet packet seqno:3 offsetInBlock:2399 lastPacketInBlock:true lastByteOffsetInBlock: 2399 \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient seqno: 3 status: SUCCESS downstreamAckTimeNanos: 0 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping org.spark-project.jetty.server.Server@489bb457 \n15/08/14 09:32:34 DEBUG DFSClient: Closing old block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping SelectChannelConnector@0.0.0.0:4040 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping org.spark-project.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@15e3d24a \n15/08/14 09:32:34 DEBUG nio: Stopped Thread[qtp1717362942-45 Selector0,5,main] on org.spark-project.jetty.io.nio.SelectorManager$1@25964fe8 \n15/08/14 09:32:34 DEBUG Client: IPC Client (123257783) connection to /9.111.251.177:9000 from root sending #6 \n15/08/14 09:32:34 DEBUG Client: IPC Client (123257783) connection to /9.111.251.177:9000 from root got value #6 \n15/08/14 09:32:34 DEBUG ProtobufRpcEngine: Call: complete took 4ms \n15/08/14 09:32:34 DEBUG nio: Stopped Thread[qtp1717362942-46 Selector1,5,main] on org.spark-project.jetty.io.nio.SelectorManager$1@2f581b9f \n15/08/14 09:32:34 DEBUG Client: stopping client from cache: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: removing client from cache: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: Stopping client \n15/08/14 09:32:34 *ERROR* LiveListenerBus: Listener EventLoggingListener threw an exception \njava.lang.reflect.InvocationTargetException \nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \nat java.lang.reflect.Method.invoke(Method.java:597) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat scala.Option.foreach(Option.scala:236) \n{noformat} \n\nFrom the stacktrace, we can see a \"Shutdown hook called\" of org.apache.spark.util.Utils, and then the shutdown hook of HDFS where DFSClient is stopped, and finally the SparkSQLEnv.stop, where the exception was thrown. \nClearly, *Spark is trying to add new event log message after HDFS client is closed, which caused the exception*. \n\nAfter checking org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver, we found that the SparkSQLEnv.stop() is invoked from a normal java runtime shutdown hook: \n{code} \n // Clean up after we exit \n Runtime.getRuntime.addShutdownHook( \n new Thread() { \n override def run() { \n SparkSQLEnv.stop() \n } \n } \n ) \n{code} \nThis shutdown hook has lower priority, and will be executed after the DFSClient is closed. \nh2. Solution \nWe changed it by *using Hadoop's ShutdownHookManager and assigning a higher priority than HDFS's shutdown hook*. It fixed the problem. \nWe'll link a pull request later for review. \nThe link is [https://github.com/apache/spark/pull/8530#issuecomment-136236733]\n\n---\n\nFrom: developer\n\nThis is fixed in 1.4 and later, I doubt we'll accept patches to 1.3 at this point in time...", "query_60": "From: reporter\nSubject: Spark-sql throws IOException on exit when using HDFS to store event log.\n\nh2. Summary \nIn Spark 1.3.1, if using HDFS to store event log, spark-sql will throw an \"java.io.IOException: Filesystem closed\" when exit. \nh2. How to reproduce \n1. Enable event log mechanism, and configure the file location to HDFS. \n You can do this by setting these two properties in spark-defaults.conf: \nspark.eventLog.enabled true \nspark.eventLog.dir hdfs://xxxxx:xxxxx/spark-events \n2. start spark-sql, and type exit once it starts. \n{noformat} \nspark-sql> exit; \n15/08/14 06:29:20 ERROR scheduler.LiveListenerBus: Listener EventLoggingListener threw an exception \nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \nat java.lang.reflect.Method.invoke(Method.java:597) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat scala.Option.foreach(Option.scala:236) \nat org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener.onApplicationEnd(EventLoggingListener.scala:181) \nat org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:54) \nat org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) \nat org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) \nat org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53) \nat org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61) \nat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1678) \nat org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60) \nCaused by: java.io.IOException: Filesystem closed \nat org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) \nat org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1985) \nat org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1946) \nat org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130) \n... 19 more \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} \n15/08/14 06:29:20 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} \n15/08/14 06:29:20 INFO ui.SparkUI: Stopped Spark web UI at http://9.111.251.177:4040 \n15/08/14 06:29:20 INFO scheduler.DAGScheduler: Stopping DAGScheduler \n15/08/14 06:29:20 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors \n15/08/14 06:29:20 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down \nException in thread \"Thread-4\" java.io.IOException: Filesystem closed \nat org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:795) \nat org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1986) \nat org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) \nat org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) \nat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) \nat org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) \nat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) \nat org.apache.spark.scheduler.EventLoggingListener.stop(EventLoggingListener.scala:198) \nat org.apache.spark.SparkContext$$anonfun$stop$4.apply(SparkContext.scala:1391) \nat org.apache.spark.SparkContext$$anonfun$stop$4.apply(SparkContext.scala:1391) \nat scala.Option.foreach(Option.scala:236) \nat org.apache.spark.SparkContext.stop(SparkContext.scala:1391) \nat org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.stop(SparkSQLEnv.scala:66) \nat org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$$anon$1.run(SparkSQLCLIDriver.scala:107) \n{noformat} \nh2. Analysis \nEnabling DEBUG level, we can see more information: \n{noformat} \n15/08/14 09:32:34 DEBUG SessionState: Removing resource dir /tmp/a9d3b5b8-3329-4e93-aee2-0a3496d661fa_resources \n15/08/14 09:32:34 DEBUG SparkSQLEnv: Shutting down Spark SQL Environment \n15/08/14 09:32:34 DEBUG DiskBlockManager: Shutdown hook called \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient writeChunk allocating new packet seqno=2, src=/spark/events/app-20150814093225-0002.snappy.inprogress, packetSize=65532, chunksPerPacket=127,bytesCurBlock=2048 \n15/08/14 09:32:34 DEBUG DFSClient: Queued packet 2 \n15/08/14 09:32:34 DEBUG DFSClient: Queued packet 3 \n15/08/14 09:32:34 DEBUG DFSClient: Waiting for ack for: 3 \n15/08/14 09:32:34 DEBUG Utils: Shutdown hook called \n15/08/14 09:32:34 DEBUG DFSClient: DataStreamer block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 sending packet packet seqno:2 offsetInBlock:2048 lastPacketInBlock:false lastByteOffsetInBlock: 2399 \n15/08/14 09:32:34 INFO SparkContext: END POST APPLICATION END \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient seqno: 2 status: SUCCESS downstreamAckTimeNanos: 0 \n15/08/14 09:32:34 DEBUG DFSClient: DataStreamer block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 sending packet packet seqno:3 offsetInBlock:2399 lastPacketInBlock:true lastByteOffsetInBlock: 2399 \n15/08/14 09:32:34 DEBUG DFSClient: DFSClient seqno: 3 status: SUCCESS downstreamAckTimeNanos: 0 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping org.spark-project.jetty.server.Server@489bb457 \n15/08/14 09:32:34 DEBUG DFSClient: Closing old block BP-456748518-127.0.0.1-1439547535043:blk_1073741833_1010 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping SelectChannelConnector@0.0.0.0:4040 \n15/08/14 09:32:34 DEBUG AbstractLifeCycle: stopping org.spark-project.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@15e3d24a \n15/08/14 09:32:34 DEBUG nio: Stopped Thread[qtp1717362942-45 Selector0,5,main] on org.spark-project.jetty.io.nio.SelectorManager$1@25964fe8 \n15/08/14 09:32:34 DEBUG Client: IPC Client (123257783) connection to /9.111.251.177:9000 from root sending #6 \n15/08/14 09:32:34 DEBUG Client: IPC Client (123257783) connection to /9.111.251.177:9000 from root got value #6 \n15/08/14 09:32:34 DEBUG ProtobufRpcEngine: Call: complete took 4ms \n15/08/14 09:32:34 DEBUG nio: Stopped Thread[qtp1717362942-46 Selector1,5,main] on org.spark-project.jetty.io.nio.SelectorManager$1@2f581b9f \n15/08/14 09:32:34 DEBUG Client: stopping client from cache: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: removing client from cache: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@3aca5e2 \n15/08/14 09:32:34 DEBUG Client: Stopping client \n15/08/14 09:32:34 *ERROR* LiveListenerBus: Listener EventLoggingListener threw an exception \njava.lang.reflect.InvocationTargetException \nat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) \nat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) \nat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) \nat java.lang.reflect.Method.invoke(Method.java:597) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144) \nat scala.Option.foreach(Option.scala:236) \n{noformat} \n\nFrom the stacktrace, we can see a \"Shutdown hook called\" of org.apache.spark.util.Utils, and then the shutdown hook of HDFS where DFSClient is stopped, and finally the SparkSQLEnv.stop, where the exception was thrown. \nClearly, *Spark is trying to add new event log message after HDFS client is closed, which caused the exception*. \n\nAfter checking org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver, we found that the SparkSQLEnv.stop() is invoked from a normal java runtime shutdown hook: \n{code} \n // Clean up after we exit \n Runtime.getRuntime.addShutdownHook( \n new Thread() { \n override def run() { \n SparkSQLEnv.stop() \n } \n } \n ) \n{code} \nThis shutdown hook has lower priority, and will be executed after the DFSClient is closed. \nh2. Solution \nWe changed it by *using Hadoop's ShutdownHookManager and assigning a higher priority than HDFS's shutdown hook*. It fixed the problem. \nWe'll link a pull request later for review. \nThe link is [https://github.com/apache/spark/pull/8530#issuecomment-136236733]\n\n---\n\nFrom: developer\n\nThis is fixed in 1.4 and later, I doubt we'll accept patches to 1.3 at this point in time...", "target_key": "SPARK-6014"} {"cluster": "JIRA-SPARK-3d34cfc41158", "issue_id": "12905775", "key": "SPARK-11173", "n_comments": 3, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: Cannot save data via MSSQL JDBC\n\nHello,\nI'm experiencing an issue in writing dataframe via JBDC. My code is\n{code:title=Example.python|borderStyle=solid}\nfrom pyspark import SparkContext\nfrom pyspark.sql import SQLContext\nimport sys\n\nsc=SparkContext(appName=\"SQL Query\")\n\nsqlctx=SQLContext(sc)\nserverName=\"SQLIPAddress\"\nserverPort=\"SQL Port\"\nserverUsername=\"username\"\nserverPassword=\"password\"\nserverDatabase=\"database\"\n#########################################################################\nconnString=\"jdbc:sqlserver://{SERVER}:{PORT};user={USER};password={PASSWORD};databasename={DATABASENAME}\"\nconnString=connString.format(SERVER=serverName,PORT=serverPort,USER=serverUsername,PASSWORD=serverPassword,DATABASENAME=serverDatabase)\n\ndf=sqlctx.read.format(\"jdbc\").options(url=connString,dbtable=\"(select * from TestTable) as test_Table\").load()\n\ndf.show()\n\ntry:\n df.write.jdbc(connString,\"Test_Target\",\"append\")\n print(\"saving completed\")\nexcept:\n print(\"Error in saving data\",sys.exc_info()[0])\n\nsc.stop()\n{code}\nEven if i specify *append*, the code throws an exception saying it is trying to create the table *Test_Target* but the table is already present.\nIf I target the script to MariaDB, all is fine\n{code:title=New Connection string|borderStyle=solid}\nconnString=\"jdbc:mysql://{SERVER}:{PORT}/{DATABASENAME}?user={USER}&password={PASSWORD}\";\n{code}\nThe problem seems to be the Microsoft JDBC driver. Can you suggest or implement same workaround?\n\nBest regards", "query_30": "From: reporter\nSubject: Cannot save data via MSSQL JDBC\n\nHello,\nI'm experiencing an issue in writing dataframe via JBDC. My code is\n{code:title=Example.python|borderStyle=solid}\nfrom pyspark import SparkContext\nfrom pyspark.sql import SQLContext\nimport sys\n\nsc=SparkContext(appName=\"SQL Query\")\n\nsqlctx=SQLContext(sc)\nserverName=\"SQLIPAddress\"\nserverPort=\"SQL Port\"\nserverUsername=\"username\"\nserverPassword=\"password\"\nserverDatabase=\"database\"\n#########################################################################\nconnString=\"jdbc:sqlserver://{SERVER}:{PORT};user={USER};password={PASSWORD};databasename={DATABASENAME}\"\nconnString=connString.format(SERVER=serverName,PORT=serverPort,USER=serverUsername,PASSWORD=serverPassword,DATABASENAME=serverDatabase)\n\ndf=sqlctx.read.format(\"jdbc\").options(url=connString,dbtable=\"(select * from TestTable) as test_Table\").load()\n\ndf.show()\n\ntry:\n df.write.jdbc(connString,\"Test_Target\",\"append\")\n print(\"saving completed\")\nexcept:\n print(\"Error in saving data\",sys.exc_info()[0])\n\nsc.stop()\n{code}\nEven if i specify *append*, the code throws an exception saying it is trying to create the table *Test_Target* but the table is already present.\nIf I target the script to MariaDB, all is fine\n{code:title=New Connection string|borderStyle=solid}\nconnString=\"jdbc:mysql://{SERVER}:{PORT}/{DATABASENAME}?user={USER}&password={PASSWORD}\";\n{code}\nThe problem seems to be the Microsoft JDBC driver. Can you suggest or implement same workaround?\n\nBest regards\n\n---\n\nFrom: developer\n\nThis does not appear to be an issue from Spark.", "query_60": "From: reporter\nSubject: Cannot save data via MSSQL JDBC\n\nHello,\nI'm experiencing an issue in writing dataframe via JBDC. My code is\n{code:title=Example.python|borderStyle=solid}\nfrom pyspark import SparkContext\nfrom pyspark.sql import SQLContext\nimport sys\n\nsc=SparkContext(appName=\"SQL Query\")\n\nsqlctx=SQLContext(sc)\nserverName=\"SQLIPAddress\"\nserverPort=\"SQL Port\"\nserverUsername=\"username\"\nserverPassword=\"password\"\nserverDatabase=\"database\"\n#########################################################################\nconnString=\"jdbc:sqlserver://{SERVER}:{PORT};user={USER};password={PASSWORD};databasename={DATABASENAME}\"\nconnString=connString.format(SERVER=serverName,PORT=serverPort,USER=serverUsername,PASSWORD=serverPassword,DATABASENAME=serverDatabase)\n\ndf=sqlctx.read.format(\"jdbc\").options(url=connString,dbtable=\"(select * from TestTable) as test_Table\").load()\n\ndf.show()\n\ntry:\n df.write.jdbc(connString,\"Test_Target\",\"append\")\n print(\"saving completed\")\nexcept:\n print(\"Error in saving data\",sys.exc_info()[0])\n\nsc.stop()\n{code}\nEven if i specify *append*, the code throws an exception saying it is trying to create the table *Test_Target* but the table is already present.\nIf I target the script to MariaDB, all is fine\n{code:title=New Connection string|borderStyle=solid}\nconnString=\"jdbc:mysql://{SERVER}:{PORT}/{DATABASENAME}?user={USER}&password={PASSWORD}\";\n{code}\nThe problem seems to be the Microsoft JDBC driver. Can you suggest or implement same workaround?\n\nBest regards\n\n---\n\nFrom: developer\n\nThis does not appear to be an issue from Spark.\n\n---\n\nFrom: developer\n\nHi,\nI suppose I've found the problem: I downloaded the source code and followed the stack trace excpetion. The exception starts in DataFrameWriter.scala, row 275 because it's trying to create a table that already exists.\n{code}\nif (!tableExists) {\n val schema = JdbcUtils.schemaString(df, url)\n val sql = s\"CREATE TABLE $table ($schema)\"\n conn.prepareStatement(sql).executeUpdate()\n }\n{code}\nIn the same file, at row 256, it's invoked \n{code}\nvar tableExists = JdbcUtils.tableExists(conn, table).\n{code}\nIn JdbcUtils.scala, the function tableExists uses the following code to check the presence of the table\n{code}\nTry(conn.prepareStatement(s\"SELECT 1 FROM $table LIMIT 1\").executeQuery().next()).isSuccess\n{code}\n_This code is not valid in SQL Server that implements *TOP* to limit the number of rows_. \nMay you change the test query in:\n{code}\nselect 1 from $table where 1=0\n{code}\nYou have an exception if the table does not exist and you don't touch the data but you're compliant with more RDMBS.\n\nThanks,\n\nBest regards", "target_key": "SPARK-9078"} {"cluster": "JIRA-SPARK-0490deb2abb9", "issue_id": "12909334", "key": "SPARK-11428", "n_comments": 3, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: Schema Merging Broken for Some Queries\n\nI have data being written into parquet format via spark streaming. The data can change slightly so schema merging is required. I load a dataframe like this\n\n{code}\nurls = [\n \"/streaming/parquet/events/key=2015-10-30*\",\n \"/streaming/parquet/events/key=2015-10-29*\"\n]\n\nsdf = sql_context.read.option(\"mergeSchema\", \"true\").parquet(*urls)\nsdf.registerTempTable('events')\n{code}\n\nIf I print the schema you can see the contested column\n\n{code}\nsdf.printSchema()\n\n\nroot\n |-- _id: string (nullable = true)\n...\n |-- d__device_s: string (nullable = true)\n |-- d__isActualPageLoad_s: string (nullable = true)\n |-- d__landing_s: string (nullable = true)\n |-- d__lang_s: string (nullable = true)\n |-- d__os_s: string (nullable = true)\n |-- d__performance_i: long (nullable = true)\n |-- d__product_s: string (nullable = true)\n |-- d__refer_s: string (nullable = true)\n |-- d__rk_i: long (nullable = true)\n |-- d__screen_s: string (nullable = true)\n |-- d__submenuName_s: string (nullable = true)\n{code}\n\nThe column that's in one but not the other file is d__product_s\n\nSo I'm able to run this query and it works fine.\n{code}\nsql_context.sql('''\n select \n distinct(d__product_s) \n from \n events\n where \n n = 'view'\n''').collect()\n\n[Row(d__product_s=u'website'),\n Row(d__product_s=u'store'),\n Row(d__product_s=None),\n Row(d__product_s=u'page')]\n\n{code}\n\nHowever if I instead use that column in the where clause things break.\n\n{code}\nsql_context.sql('''\n select \n * \n from \n events\n where \n n = 'view' and d__product_s = 'page'\n''').take(1)\n\n---------------------------------------------------------------------------\nPy4JJavaError Traceback (most recent call last)\n in ()\n 6 where\n 7 n = 'frontsite_view' and d__product_s = 'page'\n----> 8 ''').take(1)\n\n/root/spark/python/pyspark/sql/dataframe.pyc in take(self, num)\n 303 with SCCallSiteSync(self._sc) as css:\n 304 port = self._sc._jvm.org.apache.spark.sql.execution.EvaluatePython.takeAndServe(\n--> 305 self._jdf, num)\n 306 return list(_load_from_socket(port, BatchedSerializer(PickleSerializer())))\n 307 \n\n/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in __call__(self, *args)\n 536 answer = self.gateway_client.send_command(command)\n 537 return_value = get_return_value(answer, self.gateway_client,\n--> 538 self.target_id, self.name)\n 539 \n 540 for temp_arg in temp_args:\n\n/root/spark/python/pyspark/sql/utils.pyc in deco(*a, **kw)\n 34 def deco(*a, **kw):\n 35 try:\n---> 36 return f(*a, **kw)\n 37 except py4j.protocol.Py4JJavaError as e:\n 38 s = e.java_exception.toString()\n\n/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)\n 298 raise Py4JJavaError(\n 299 'An error occurred while calling {0}{1}{2}.\\n'.\n--> 300 format(target_id, '.', name), value)\n 301 else:\n 302 raise Py4JError(\n\nPy4JJavaError: An error occurred while calling z:org.apache.spark.sql.execution.EvaluatePython.takeAndServe.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 30 times, most recent failure: Lost task 0.29 in stage 15.0 (TID 6536, 10.X.X.X): java.lang.IllegalArgumentException: Column [d__product_s] was not found in schema!\n\tat org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:190)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:178)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:94)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:180)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:131)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$And.accept(Operators.java:308)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)\n\tat org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:160)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:155)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:120)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)\n\tat scala.Option.foreach(Option.scala:236)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1835)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1848)\n\tat org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:215)\n\tat org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:207)\n\tat org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385)\n\tat org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)\n\tat org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1903)\n\tat org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1384)\n\tat org.apache.spark.sql.DataFrame.head(DataFrame.scala:1314)\n\tat org.apache.spark.sql.DataFrame.take(DataFrame.scala:1377)\n\tat org.apache.spark.sql.execution.EvaluatePython$.takeAndServe(python.scala:127)\n\tat org.apache.spark.sql.execution.EvaluatePython.takeAndServe(python.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:606)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)\n\tat py4j.Gateway.invoke(Gateway.java:259)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:207)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.IllegalArgumentException: Column [d__product_s] was not found in schema!\n\tat org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:190)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:178)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:94)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:180)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:131)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$And.accept(Operators.java:308)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)\n\tat org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:160)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:155)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:120)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\t... 1 more\n\n{code}\n\nI get the same error also when attempting to write the same query with the dataframe api as well.\n\n{code}\nsdf.where(sdf.d__product_s == 'page').take(1)\n{code}", "query_30": "From: reporter\nSubject: Schema Merging Broken for Some Queries\n\nI have data being written into parquet format via spark streaming. The data can change slightly so schema merging is required. I load a dataframe like this\n\n{code}\nurls = [\n \"/streaming/parquet/events/key=2015-10-30*\",\n \"/streaming/parquet/events/key=2015-10-29*\"\n]\n\nsdf = sql_context.read.option(\"mergeSchema\", \"true\").parquet(*urls)\nsdf.registerTempTable('events')\n{code}\n\nIf I print the schema you can see the contested column\n\n{code}\nsdf.printSchema()\n\n\nroot\n |-- _id: string (nullable = true)\n...\n |-- d__device_s: string (nullable = true)\n |-- d__isActualPageLoad_s: string (nullable = true)\n |-- d__landing_s: string (nullable = true)\n |-- d__lang_s: string (nullable = true)\n |-- d__os_s: string (nullable = true)\n |-- d__performance_i: long (nullable = true)\n |-- d__product_s: string (nullable = true)\n |-- d__refer_s: string (nullable = true)\n |-- d__rk_i: long (nullable = true)\n |-- d__screen_s: string (nullable = true)\n |-- d__submenuName_s: string (nullable = true)\n{code}\n\nThe column that's in one but not the other file is d__product_s\n\nSo I'm able to run this query and it works fine.\n{code}\nsql_context.sql('''\n select \n distinct(d__product_s) \n from \n events\n where \n n = 'view'\n''').collect()\n\n[Row(d__product_s=u'website'),\n Row(d__product_s=u'store'),\n Row(d__product_s=None),\n Row(d__product_s=u'page')]\n\n{code}\n\nHowever if I instead use that column in the where clause things break.\n\n{code}\nsql_context.sql('''\n select \n * \n from \n events\n where \n n = 'view' and d__product_s = 'page'\n''').take(1)\n\n---------------------------------------------------------------------------\nPy4JJavaError Traceback (most recent call last)\n in ()\n 6 where\n 7 n = 'frontsite_view' and d__product_s = 'page'\n----> 8 ''').take(1)\n\n/root/spark/python/pyspark/sql/dataframe.pyc in take(self, num)\n 303 with SCCallSiteSync(self._sc) as css:\n 304 port = self._sc._jvm.org.apache.spark.sql.execution.EvaluatePython.takeAndServe(\n--> 305 self._jdf, num)\n 306 return list(_load_from_socket(port, BatchedSerializer(PickleSerializer())))\n 307 \n\n/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in __call__(self, *args)\n 536 answer = self.gateway_client.send_command(command)\n 537 return_value = get_return_value(answer, self.gateway_client,\n--> 538 self.target_id, self.name)\n 539 \n 540 for temp_arg in temp_args:\n\n/root/spark/python/pyspark/sql/utils.pyc in deco(*a, **kw)\n 34 def deco(*a, **kw):\n 35 try:\n---> 36 return f(*a, **kw)\n 37 except py4j.protocol.Py4JJavaError as e:\n 38 s = e.java_exception.toString()\n\n/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)\n 298 raise Py4JJavaError(\n 299 'An error occurred while calling {0}{1}{2}.\\n'.\n--> 300 format(target_id, '.', name), value)\n 301 else:\n 302 raise Py4JError(\n\nPy4JJavaError: An error occurred while calling z:org.apache.spark.sql.execution.EvaluatePython.takeAndServe.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 30 times, most recent failure: Lost task 0.29 in stage 15.0 (TID 6536, 10.X.X.X): java.lang.IllegalArgumentException: Column [d__product_s] was not found in schema!\n\tat org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:190)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:178)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:94)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:180)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:131)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$And.accept(Operators.java:308)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)\n\tat org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:160)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:155)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:120)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)\n\tat scala.Option.foreach(Option.scala:236)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1835)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1848)\n\tat org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:215)\n\tat org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:207)\n\tat org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385)\n\tat org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)\n\tat org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1903)\n\tat org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1384)\n\tat org.apache.spark.sql.DataFrame.head(DataFrame.scala:1314)\n\tat org.apache.spark.sql.DataFrame.take(DataFrame.scala:1377)\n\tat org.apache.spark.sql.execution.EvaluatePython$.takeAndServe(python.scala:127)\n\tat org.apache.spark.sql.execution.EvaluatePython.takeAndServe(python.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:606)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)\n\tat py4j.Gateway.invoke(Gateway.java:259)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:207)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.IllegalArgumentException: Column [d__product_s] was not found in schema!\n\tat org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:190)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:178)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:94)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:180)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:131)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$And.accept(Operators.java:308)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)\n\tat org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:160)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:155)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:120)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\t... 1 more\n\n{code}\n\nI get the same error also when attempting to write the same query with the dataframe api as well.\n\n{code}\nsdf.where(sdf.d__product_s == 'page').take(1)\n{code}\n\n---\n\nFrom: developer\n\nRelated to https://issues.apache.org/jira/browse/SPARK-11412 ?", "query_60": "From: reporter\nSubject: Schema Merging Broken for Some Queries\n\nI have data being written into parquet format via spark streaming. The data can change slightly so schema merging is required. I load a dataframe like this\n\n{code}\nurls = [\n \"/streaming/parquet/events/key=2015-10-30*\",\n \"/streaming/parquet/events/key=2015-10-29*\"\n]\n\nsdf = sql_context.read.option(\"mergeSchema\", \"true\").parquet(*urls)\nsdf.registerTempTable('events')\n{code}\n\nIf I print the schema you can see the contested column\n\n{code}\nsdf.printSchema()\n\n\nroot\n |-- _id: string (nullable = true)\n...\n |-- d__device_s: string (nullable = true)\n |-- d__isActualPageLoad_s: string (nullable = true)\n |-- d__landing_s: string (nullable = true)\n |-- d__lang_s: string (nullable = true)\n |-- d__os_s: string (nullable = true)\n |-- d__performance_i: long (nullable = true)\n |-- d__product_s: string (nullable = true)\n |-- d__refer_s: string (nullable = true)\n |-- d__rk_i: long (nullable = true)\n |-- d__screen_s: string (nullable = true)\n |-- d__submenuName_s: string (nullable = true)\n{code}\n\nThe column that's in one but not the other file is d__product_s\n\nSo I'm able to run this query and it works fine.\n{code}\nsql_context.sql('''\n select \n distinct(d__product_s) \n from \n events\n where \n n = 'view'\n''').collect()\n\n[Row(d__product_s=u'website'),\n Row(d__product_s=u'store'),\n Row(d__product_s=None),\n Row(d__product_s=u'page')]\n\n{code}\n\nHowever if I instead use that column in the where clause things break.\n\n{code}\nsql_context.sql('''\n select \n * \n from \n events\n where \n n = 'view' and d__product_s = 'page'\n''').take(1)\n\n---------------------------------------------------------------------------\nPy4JJavaError Traceback (most recent call last)\n in ()\n 6 where\n 7 n = 'frontsite_view' and d__product_s = 'page'\n----> 8 ''').take(1)\n\n/root/spark/python/pyspark/sql/dataframe.pyc in take(self, num)\n 303 with SCCallSiteSync(self._sc) as css:\n 304 port = self._sc._jvm.org.apache.spark.sql.execution.EvaluatePython.takeAndServe(\n--> 305 self._jdf, num)\n 306 return list(_load_from_socket(port, BatchedSerializer(PickleSerializer())))\n 307 \n\n/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py in __call__(self, *args)\n 536 answer = self.gateway_client.send_command(command)\n 537 return_value = get_return_value(answer, self.gateway_client,\n--> 538 self.target_id, self.name)\n 539 \n 540 for temp_arg in temp_args:\n\n/root/spark/python/pyspark/sql/utils.pyc in deco(*a, **kw)\n 34 def deco(*a, **kw):\n 35 try:\n---> 36 return f(*a, **kw)\n 37 except py4j.protocol.Py4JJavaError as e:\n 38 s = e.java_exception.toString()\n\n/root/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)\n 298 raise Py4JJavaError(\n 299 'An error occurred while calling {0}{1}{2}.\\n'.\n--> 300 format(target_id, '.', name), value)\n 301 else:\n 302 raise Py4JError(\n\nPy4JJavaError: An error occurred while calling z:org.apache.spark.sql.execution.EvaluatePython.takeAndServe.\n: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 30 times, most recent failure: Lost task 0.29 in stage 15.0 (TID 6536, 10.X.X.X): java.lang.IllegalArgumentException: Column [d__product_s] was not found in schema!\n\tat org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:190)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:178)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:94)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:180)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:131)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$And.accept(Operators.java:308)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)\n\tat org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:160)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:155)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:120)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\n\nDriver stacktrace:\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)\n\tat scala.Option.foreach(Option.scala:236)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1835)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1848)\n\tat org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:215)\n\tat org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:207)\n\tat org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385)\n\tat org.apache.spark.sql.DataFrame$$anonfun$collect$1.apply(DataFrame.scala:1385)\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)\n\tat org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:1903)\n\tat org.apache.spark.sql.DataFrame.collect(DataFrame.scala:1384)\n\tat org.apache.spark.sql.DataFrame.head(DataFrame.scala:1314)\n\tat org.apache.spark.sql.DataFrame.take(DataFrame.scala:1377)\n\tat org.apache.spark.sql.execution.EvaluatePython$.takeAndServe(python.scala:127)\n\tat org.apache.spark.sql.execution.EvaluatePython.takeAndServe(python.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:606)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)\n\tat py4j.Gateway.invoke(Gateway.java:259)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:207)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.IllegalArgumentException: Column [d__product_s] was not found in schema!\n\tat org.apache.parquet.Preconditions.checkArgument(Preconditions.java:55)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:190)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:178)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:160)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:94)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$Eq.accept(Operators.java:180)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:131)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:59)\n\tat org.apache.parquet.filter2.predicate.Operators$And.accept(Operators.java:308)\n\tat org.apache.parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:64)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:59)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:40)\n\tat org.apache.parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:126)\n\tat org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:46)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:160)\n\tat org.apache.parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:140)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD$$anon$1.(SqlNewHadoopRDD.scala:155)\n\tat org.apache.spark.rdd.SqlNewHadoopRDD.compute(SqlNewHadoopRDD.scala:120)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\n\tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)\n\tat org.apache.spark.rdd.RDD.iterator(RDD.scala:264)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:88)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\t... 1 more\n\n{code}\n\nI get the same error also when attempting to write the same query with the dataframe api as well.\n\n{code}\nsdf.where(sdf.d__product_s == 'page').take(1)\n{code}\n\n---\n\nFrom: developer\n\nRelated to https://issues.apache.org/jira/browse/SPARK-11412 ?\n\n---\n\nFrom: developer\n\nAs a work around. You can create a new parquet file from the schema merged data frame and then run the query on the merged parquet file. However it's not practical for large datasets and really should be fixed.", "target_key": "SPARK-11103"} {"cluster": "JIRA-SPARK-d5e7a51e2bf0", "issue_id": "12919563", "key": "SPARK-12167", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "SPARK", "query_0": "From: reporter\nSubject: Invoke the right sameResult function when plan is warpped with SubQueries\n\nI find this bug when I use cache table,\n```\nspark-sql> create table src_p(key int, value int) stored as parquet;\nOK\nTime taken: 3.144 seconds\nspark-sql> cache table src_p;\nTime taken: 1.452 seconds\nspark-sql> explain extended select count(*) from src_p;\n```\nI got the wrong physical plan\n```\n== Physical Plan ==\nTungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#28L])\n TungstenExchange SinglePartition\n TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#33L])\n Scan ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p][]\n```\nand the right physical plan is\n```\n== Physical Plan ==\nTungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#47L])\n TungstenExchange SinglePartition\n TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#62L])\n InMemoryColumnarTableScan (InMemoryRelation [key#45,value#46], true, 10000, StorageLevel(true, true, false, true, 1), (Scan ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p][key#9,value#10]), Some(src_p))\n```\n\nWhen the implementation classes of `MultiInstanceRelation`(eg. `LogicalRelation`, `LocalRelation`) are warpped with SubQueries, they can't invoke the right `sameResult` function in their own implementation. So we need to eliminate SubQueries first and then try to invoke `sameResult` function in their own implementation.\nLike:\nWhen plan is `Subquery(LogicalRelation(relation:ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p], expectedOutputAttributes:Some(ArrayBuffer(key#0, value#1))))`, first eliminate SubQueries, and then will invoke the `sameResult` function in `LogicalRelation` instead of `LogicalPlan`.", "query_30": "From: reporter\nSubject: Invoke the right sameResult function when plan is warpped with SubQueries\n\nI find this bug when I use cache table,\n```\nspark-sql> create table src_p(key int, value int) stored as parquet;\nOK\nTime taken: 3.144 seconds\nspark-sql> cache table src_p;\nTime taken: 1.452 seconds\nspark-sql> explain extended select count(*) from src_p;\n```\nI got the wrong physical plan\n```\n== Physical Plan ==\nTungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#28L])\n TungstenExchange SinglePartition\n TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#33L])\n Scan ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p][]\n```\nand the right physical plan is\n```\n== Physical Plan ==\nTungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#47L])\n TungstenExchange SinglePartition\n TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#62L])\n InMemoryColumnarTableScan (InMemoryRelation [key#45,value#46], true, 10000, StorageLevel(true, true, false, true, 1), (Scan ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p][key#9,value#10]), Some(src_p))\n```\n\nWhen the implementation classes of `MultiInstanceRelation`(eg. `LogicalRelation`, `LocalRelation`) are warpped with SubQueries, they can't invoke the right `sameResult` function in their own implementation. So we need to eliminate SubQueries first and then try to invoke `sameResult` function in their own implementation.\nLike:\nWhen plan is `Subquery(LogicalRelation(relation:ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p], expectedOutputAttributes:Some(ArrayBuffer(key#0, value#1))))`, first eliminate SubQueries, and then will invoke the `sameResult` function in `LogicalRelation` instead of `LogicalPlan`.\n\n---\n\nFrom: developer\n\nUser 'watermen' has created a pull request for this issue:\nhttps://github.com/apache/spark/pull/10169", "query_60": "From: reporter\nSubject: Invoke the right sameResult function when plan is warpped with SubQueries\n\nI find this bug when I use cache table,\n```\nspark-sql> create table src_p(key int, value int) stored as parquet;\nOK\nTime taken: 3.144 seconds\nspark-sql> cache table src_p;\nTime taken: 1.452 seconds\nspark-sql> explain extended select count(*) from src_p;\n```\nI got the wrong physical plan\n```\n== Physical Plan ==\nTungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#28L])\n TungstenExchange SinglePartition\n TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#33L])\n Scan ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p][]\n```\nand the right physical plan is\n```\n== Physical Plan ==\nTungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#47L])\n TungstenExchange SinglePartition\n TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[currentCount#62L])\n InMemoryColumnarTableScan (InMemoryRelation [key#45,value#46], true, 10000, StorageLevel(true, true, false, true, 1), (Scan ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p][key#9,value#10]), Some(src_p))\n```\n\nWhen the implementation classes of `MultiInstanceRelation`(eg. `LogicalRelation`, `LocalRelation`) are warpped with SubQueries, they can't invoke the right `sameResult` function in their own implementation. So we need to eliminate SubQueries first and then try to invoke `sameResult` function in their own implementation.\nLike:\nWhen plan is `Subquery(LogicalRelation(relation:ParquetRelation[hdfs://9.91.8.131:9000/user/hive/warehouse/src_p], expectedOutputAttributes:Some(ArrayBuffer(key#0, value#1))))`, first eliminate SubQueries, and then will invoke the `sameResult` function in `LogicalRelation` instead of `LogicalPlan`.\n\n---\n\nFrom: developer\n\nUser 'watermen' has created a pull request for this issue:\nhttps://github.com/apache/spark/pull/10169", "target_key": "SPARK-11246"} {"cluster": "JIRA-SPARK-608f77bb30b3", "issue_id": "12923863", "key": "SPARK-12491", "n_comments": 7, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: UDAF result differs in SQL if alias is used\n\nUsing the GeometricMean UDAF example (https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html), I found the following discrepancy in results:\n\n{code}\nscala> sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\").show()\n+--------+---+\n|group_id|_c1|\n+--------+---+\n| 0|0.0|\n| 1|0.0|\n| 2|0.0|\n+--------+---+\n\n\nscala> sqlContext.sql(\"select group_id, gm(id) as GeometricMean from simple group by group_id\").show()\n+--------+-----------------+\n|group_id| GeometricMean|\n+--------+-----------------+\n| 0|8.981385496571725|\n| 1|7.301716979342118|\n| 2|7.706253151292568|\n+--------+-----------------+\n{code}", "query_30": "From: reporter\nSubject: UDAF result differs in SQL if alias is used\n\nUsing the GeometricMean UDAF example (https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html), I found the following discrepancy in results:\n\n{code}\nscala> sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\").show()\n+--------+---+\n|group_id|_c1|\n+--------+---+\n| 0|0.0|\n| 1|0.0|\n| 2|0.0|\n+--------+---+\n\n\nscala> sqlContext.sql(\"select group_id, gm(id) as GeometricMean from simple group by group_id\").show()\n+--------+-----------------+\n|group_id| GeometricMean|\n+--------+-----------------+\n| 0|8.981385496571725|\n| 1|7.301716979342118|\n| 2|7.706253151292568|\n+--------+-----------------+\n{code}\n\n---\n\nFrom: developer\n\nI cannot reproduce this issue on 1.5.2 and the latest master in local mode.\n\nI used the following code:\n\n{noformat}\nimport org.apache.spark.sql.expressions.MutableAggregationBuffer\nimport org.apache.spark.sql.expressions.UserDefinedAggregateFunction\nimport org.apache.spark.sql.Row\nimport org.apache.spark.sql.types._\n\nclass GeometricMean extends UserDefinedAggregateFunction {\n def inputSchema: org.apache.spark.sql.types.StructType =\n StructType(StructField(\"value\", DoubleType) :: Nil)\n\n def bufferSchema: StructType = StructType(\n StructField(\"count\", LongType) ::\n StructField(\"product\", DoubleType) :: Nil\n )\n\n def dataType: DataType = DoubleType\n\n def deterministic: Boolean = true\n\n def initialize(buffer: MutableAggregationBuffer): Unit = {\n buffer(0) = 0L\n buffer(1) = 1.0\n }\n\n def update(buffer: MutableAggregationBuffer,input: Row): Unit = {\n buffer(0) = buffer.getAs[Long](0) + 1\n buffer(1) = buffer.getAs[Double](1) * input.getAs[Double](0)\n }\n\n def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {\n buffer1(0) = buffer1.getAs[Long](0) + buffer2.getAs[Long](0)\n buffer1(1) = buffer1.getAs[Double](1) * buffer2.getAs[Double](1)\n }\n\n def evaluate(buffer: Row): Any = {\n math.pow(buffer.getDouble(1), 1.toDouble / buffer.getLong(0))\n }\n}\n\n// Create an instance of UDAF GeometricMean.\nval gm = new GeometricMean\n\n// Register the udaf\nsqlContext.udf.register(\"gm\", gm)\n\n// Create a simple DataFrame with a single column called \"id\"\n// containing number 1 to 10.\nval df = sqlContext.range(1, 11).select(($\"id\" / 3).cast(\"int\").as(\"group_id\"), $\"id\")\ndf.registerTempTable(\"simple\")\n\n// Without alias\nsqlContext.sql(\"select group_id, gm(id) from simple group by group_id\").show()\n\n// With alias\nsqlContext.sql(\"select group_id, gm(id) as GeometricMean from simple group by group_id\").show()\n{noformat}\n\nCould you share the logical plans for both queries? You can get a logical plan by doing this:\n\n{noformat}\n// Query without alias\nval q = sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\")\nq.explain(true)\n{noformat}\n\n\n---\n\nFrom: developer\n\nREPL with logical plans:\n\n{code:None|borderStyle=solid}\nscala> import com.pipeline.spark._\nimport com.pipeline.spark._\n\nscala> sqlContext.udf.register(\"gm\", new GeometricMean)\nres0: org.apache.spark.sql.expressions.UserDefinedAggregateFunction = com.pipeline.spark.GeometricMean@497031ea\n\nscala> import org.apache.spark.sql.functions._\nimport org.apache.spark.sql.functions._\n\nscala> val ids = sqlContext.range(1, 20)\nids: org.apache.spark.sql.DataFrame = [id: bigint]\n\nscala> ids.registerTempTable(\"ids\")\n\nscala> val df = sqlContext.sql(\"select id, id % 3 as group_id from ids\")\ndf: org.apache.spark.sql.DataFrame = [id: bigint, group_id: bigint]\n\nscala> df.registerTempTable(\"simple\")\n\nscala> val q = sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\")\nq: org.apache.spark.sql.DataFrame = [group_id: bigint, _c1: double]\n\nscala> q.explain(true)\n== Parsed Logical Plan ==\n'Aggregate ['group_id], [unresolvedalias('group_id),unresolvedalias('gm('id))]\n 'UnresolvedRelation [simple], None\n\n== Analyzed Logical Plan ==\ngroup_id: bigint, _c1: double\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS _c1#12]\n Subquery simple\n Project [id#0L,(id#0L % cast(3 as bigint)) AS group_id#1L]\n Subquery ids\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Optimized Logical Plan ==\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS _c1#12]\n Project [id#0L,(id#0L % 3) AS group_id#1L]\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Physical Plan ==\nSortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Final,isDistinct=false)], output=[group_id#1L,_c1#12])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenExchange hashpartitioning(group_id#1L)\n ConvertToUnsafe\n SortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Partial,isDistinct=false)], output=[group_id#1L,count#14L,product#15])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenProject [id#0L,(id#0L % 3) AS group_id#1L]\n Scan PhysicalRDD[id#0L]\n\nCode Generation: true\n\nscala> q.show()\n+--------+---+\n|group_id|_c1|\n+--------+---+\n| 0|0.0|\n| 1|0.0|\n| 2|0.0|\n+--------+---+\n\n\nscala> val q2 = sqlContext.sql(\"select group_id, gm(id) as geomean from simple group by group_id\")\nq2: org.apache.spark.sql.DataFrame = [group_id: bigint, geomean: double]\n\nscala> q2.explain(true)\n== Parsed Logical Plan ==\n'Aggregate ['group_id], [unresolvedalias('group_id),unresolvedalias('gm('id) AS geomean#19)]\n 'UnresolvedRelation [simple], None\n\n== Analyzed Logical Plan ==\ngroup_id: bigint, geomean: double\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS geomean#19]\n Subquery simple\n Project [id#0L,(id#0L % cast(3 as bigint)) AS group_id#1L]\n Subquery ids\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Optimized Logical Plan ==\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS geomean#19]\n Project [id#0L,(id#0L % 3) AS group_id#1L]\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Physical Plan ==\nSortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Final,isDistinct=false)], output=[group_id#1L,geomean#19])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenExchange hashpartitioning(group_id#1L)\n ConvertToUnsafe\n SortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Partial,isDistinct=false)], output=[group_id#1L,count#30L,product#31])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenProject [id#0L,(id#0L % 3) AS group_id#1L]\n Scan PhysicalRDD[id#0L]\n\nCode Generation: true\n\n \nscala> q2.show()\n+--------+-----------------+\n|group_id| geomean|\n+--------+-----------------+\n| 0|8.981385496571725|\n| 1|7.301716979342118|\n| 2|7.706253151292568|\n+--------+-----------------+\n{code}\n\nAnd here is the UDAF spec:\n\n{code:None|borderStyle=solid}\npackage com.pipeline.spark\n\nimport org.apache.spark.sql.expressions.MutableAggregationBuffer\nimport org.apache.spark.sql.expressions.UserDefinedAggregateFunction\nimport org.apache.spark.sql.Row\nimport org.apache.spark.sql.types._\n\n\nclass GeometricMean extends UserDefinedAggregateFunction {\n // This is the input fields for your aggregate function.\n def inputSchema: org.apache.spark.sql.types.StructType = \n StructType(StructField(\"value\", DoubleType) :: Nil)\n\n // This is the internal fields you keep for computing your aggregate.\n def bufferSchema: StructType = StructType(\n StructField(\"count\", LongType) ::\n StructField(\"product\", DoubleType) :: Nil\n )\n \n // This is the output type of your aggregatation function.\n def dataType: DataType = DoubleType\n \n def deterministic: Boolean = true\n \n // This is the initial value for your buffer schema.\n def initialize(buffer: MutableAggregationBuffer): Unit = {\n buffer(0) = 0L\n buffer(1) = 1.0\n }\n \n // This is how to update your buffer schema given an input.\n def update(buffer: MutableAggregationBuffer, input: Row): Unit = {\n buffer(0) = buffer.getAs[Long](0) + 1\n buffer(1) = buffer.getAs[Double](1) * input.getAs[Double](0)\n }\n \n // This is how to merge two objects with the bufferSchema type.\n def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {\n buffer1(0) = buffer1.getAs[Long](0) + buffer2.getAs[Long](0)\n buffer1(1) = buffer1.getAs[Double](1) * buffer2.getAs[Double](1)\n }\n\n // This is where you output the final value, given the final value of your bufferSchema.\n def evaluate(buffer: Row): Any = {\n math.pow(buffer.getDouble(1), 1.toDouble / buffer.getLong(0))\n }\n}\n{code}\n\n---\n\nFrom: developer\n\nThe logical plans look fine.\n\nI have defined the class in a separate project, create an assembly, {{gm.jar}}, from that, and finally start the the spark-shell using the following command: {{./spark-shell --master local\\[\\*\\] --driver-memory 10G --jars gm.jar}}\n\nUnfortunately I still cannot reproduce your problem (I'll attach the project).\n\nAre you doing this in local mode or cluster mode? What kind of cluster are you using? What deps are in jars you pass to spark?\n\n", "query_60": "From: reporter\nSubject: UDAF result differs in SQL if alias is used\n\nUsing the GeometricMean UDAF example (https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html), I found the following discrepancy in results:\n\n{code}\nscala> sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\").show()\n+--------+---+\n|group_id|_c1|\n+--------+---+\n| 0|0.0|\n| 1|0.0|\n| 2|0.0|\n+--------+---+\n\n\nscala> sqlContext.sql(\"select group_id, gm(id) as GeometricMean from simple group by group_id\").show()\n+--------+-----------------+\n|group_id| GeometricMean|\n+--------+-----------------+\n| 0|8.981385496571725|\n| 1|7.301716979342118|\n| 2|7.706253151292568|\n+--------+-----------------+\n{code}\n\n---\n\nFrom: developer\n\nI cannot reproduce this issue on 1.5.2 and the latest master in local mode.\n\nI used the following code:\n\n{noformat}\nimport org.apache.spark.sql.expressions.MutableAggregationBuffer\nimport org.apache.spark.sql.expressions.UserDefinedAggregateFunction\nimport org.apache.spark.sql.Row\nimport org.apache.spark.sql.types._\n\nclass GeometricMean extends UserDefinedAggregateFunction {\n def inputSchema: org.apache.spark.sql.types.StructType =\n StructType(StructField(\"value\", DoubleType) :: Nil)\n\n def bufferSchema: StructType = StructType(\n StructField(\"count\", LongType) ::\n StructField(\"product\", DoubleType) :: Nil\n )\n\n def dataType: DataType = DoubleType\n\n def deterministic: Boolean = true\n\n def initialize(buffer: MutableAggregationBuffer): Unit = {\n buffer(0) = 0L\n buffer(1) = 1.0\n }\n\n def update(buffer: MutableAggregationBuffer,input: Row): Unit = {\n buffer(0) = buffer.getAs[Long](0) + 1\n buffer(1) = buffer.getAs[Double](1) * input.getAs[Double](0)\n }\n\n def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {\n buffer1(0) = buffer1.getAs[Long](0) + buffer2.getAs[Long](0)\n buffer1(1) = buffer1.getAs[Double](1) * buffer2.getAs[Double](1)\n }\n\n def evaluate(buffer: Row): Any = {\n math.pow(buffer.getDouble(1), 1.toDouble / buffer.getLong(0))\n }\n}\n\n// Create an instance of UDAF GeometricMean.\nval gm = new GeometricMean\n\n// Register the udaf\nsqlContext.udf.register(\"gm\", gm)\n\n// Create a simple DataFrame with a single column called \"id\"\n// containing number 1 to 10.\nval df = sqlContext.range(1, 11).select(($\"id\" / 3).cast(\"int\").as(\"group_id\"), $\"id\")\ndf.registerTempTable(\"simple\")\n\n// Without alias\nsqlContext.sql(\"select group_id, gm(id) from simple group by group_id\").show()\n\n// With alias\nsqlContext.sql(\"select group_id, gm(id) as GeometricMean from simple group by group_id\").show()\n{noformat}\n\nCould you share the logical plans for both queries? You can get a logical plan by doing this:\n\n{noformat}\n// Query without alias\nval q = sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\")\nq.explain(true)\n{noformat}\n\n\n---\n\nFrom: developer\n\nREPL with logical plans:\n\n{code:None|borderStyle=solid}\nscala> import com.pipeline.spark._\nimport com.pipeline.spark._\n\nscala> sqlContext.udf.register(\"gm\", new GeometricMean)\nres0: org.apache.spark.sql.expressions.UserDefinedAggregateFunction = com.pipeline.spark.GeometricMean@497031ea\n\nscala> import org.apache.spark.sql.functions._\nimport org.apache.spark.sql.functions._\n\nscala> val ids = sqlContext.range(1, 20)\nids: org.apache.spark.sql.DataFrame = [id: bigint]\n\nscala> ids.registerTempTable(\"ids\")\n\nscala> val df = sqlContext.sql(\"select id, id % 3 as group_id from ids\")\ndf: org.apache.spark.sql.DataFrame = [id: bigint, group_id: bigint]\n\nscala> df.registerTempTable(\"simple\")\n\nscala> val q = sqlContext.sql(\"select group_id, gm(id) from simple group by group_id\")\nq: org.apache.spark.sql.DataFrame = [group_id: bigint, _c1: double]\n\nscala> q.explain(true)\n== Parsed Logical Plan ==\n'Aggregate ['group_id], [unresolvedalias('group_id),unresolvedalias('gm('id))]\n 'UnresolvedRelation [simple], None\n\n== Analyzed Logical Plan ==\ngroup_id: bigint, _c1: double\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS _c1#12]\n Subquery simple\n Project [id#0L,(id#0L % cast(3 as bigint)) AS group_id#1L]\n Subquery ids\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Optimized Logical Plan ==\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS _c1#12]\n Project [id#0L,(id#0L % 3) AS group_id#1L]\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Physical Plan ==\nSortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Final,isDistinct=false)], output=[group_id#1L,_c1#12])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenExchange hashpartitioning(group_id#1L)\n ConvertToUnsafe\n SortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Partial,isDistinct=false)], output=[group_id#1L,count#14L,product#15])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenProject [id#0L,(id#0L % 3) AS group_id#1L]\n Scan PhysicalRDD[id#0L]\n\nCode Generation: true\n\nscala> q.show()\n+--------+---+\n|group_id|_c1|\n+--------+---+\n| 0|0.0|\n| 1|0.0|\n| 2|0.0|\n+--------+---+\n\n\nscala> val q2 = sqlContext.sql(\"select group_id, gm(id) as geomean from simple group by group_id\")\nq2: org.apache.spark.sql.DataFrame = [group_id: bigint, geomean: double]\n\nscala> q2.explain(true)\n== Parsed Logical Plan ==\n'Aggregate ['group_id], [unresolvedalias('group_id),unresolvedalias('gm('id) AS geomean#19)]\n 'UnresolvedRelation [simple], None\n\n== Analyzed Logical Plan ==\ngroup_id: bigint, geomean: double\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS geomean#19]\n Subquery simple\n Project [id#0L,(id#0L % cast(3 as bigint)) AS group_id#1L]\n Subquery ids\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Optimized Logical Plan ==\nAggregate [group_id#1L], [group_id#1L,(GeometricMean(cast(id#0L as double)),mode=Complete,isDistinct=false) AS geomean#19]\n Project [id#0L,(id#0L % 3) AS group_id#1L]\n LogicalRDD [id#0L], MapPartitionsRDD[3] at range at :25\n\n== Physical Plan ==\nSortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Final,isDistinct=false)], output=[group_id#1L,geomean#19])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenExchange hashpartitioning(group_id#1L)\n ConvertToUnsafe\n SortBasedAggregate(key=[group_id#1L], functions=[(GeometricMean(cast(id#0L as double)),mode=Partial,isDistinct=false)], output=[group_id#1L,count#30L,product#31])\n ConvertToSafe\n TungstenSort [group_id#1L ASC], false, 0\n TungstenProject [id#0L,(id#0L % 3) AS group_id#1L]\n Scan PhysicalRDD[id#0L]\n\nCode Generation: true\n\n \nscala> q2.show()\n+--------+-----------------+\n|group_id| geomean|\n+--------+-----------------+\n| 0|8.981385496571725|\n| 1|7.301716979342118|\n| 2|7.706253151292568|\n+--------+-----------------+\n{code}\n\nAnd here is the UDAF spec:\n\n{code:None|borderStyle=solid}\npackage com.pipeline.spark\n\nimport org.apache.spark.sql.expressions.MutableAggregationBuffer\nimport org.apache.spark.sql.expressions.UserDefinedAggregateFunction\nimport org.apache.spark.sql.Row\nimport org.apache.spark.sql.types._\n\n\nclass GeometricMean extends UserDefinedAggregateFunction {\n // This is the input fields for your aggregate function.\n def inputSchema: org.apache.spark.sql.types.StructType = \n StructType(StructField(\"value\", DoubleType) :: Nil)\n\n // This is the internal fields you keep for computing your aggregate.\n def bufferSchema: StructType = StructType(\n StructField(\"count\", LongType) ::\n StructField(\"product\", DoubleType) :: Nil\n )\n \n // This is the output type of your aggregatation function.\n def dataType: DataType = DoubleType\n \n def deterministic: Boolean = true\n \n // This is the initial value for your buffer schema.\n def initialize(buffer: MutableAggregationBuffer): Unit = {\n buffer(0) = 0L\n buffer(1) = 1.0\n }\n \n // This is how to update your buffer schema given an input.\n def update(buffer: MutableAggregationBuffer, input: Row): Unit = {\n buffer(0) = buffer.getAs[Long](0) + 1\n buffer(1) = buffer.getAs[Double](1) * input.getAs[Double](0)\n }\n \n // This is how to merge two objects with the bufferSchema type.\n def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {\n buffer1(0) = buffer1.getAs[Long](0) + buffer2.getAs[Long](0)\n buffer1(1) = buffer1.getAs[Double](1) * buffer2.getAs[Double](1)\n }\n\n // This is where you output the final value, given the final value of your bufferSchema.\n def evaluate(buffer: Row): Any = {\n math.pow(buffer.getDouble(1), 1.toDouble / buffer.getLong(0))\n }\n}\n{code}\n\n---\n\nFrom: developer\n\nThe logical plans look fine.\n\nI have defined the class in a separate project, create an assembly, {{gm.jar}}, from that, and finally start the the spark-shell using the following command: {{./spark-shell --master local\\[\\*\\] --driver-memory 10G --jars gm.jar}}\n\nUnfortunately I still cannot reproduce your problem (I'll attach the project).\n\nAre you doing this in local mode or cluster mode? What kind of cluster are you using? What deps are in jars you pass to spark?\n\n\n\n---\n\nFrom: developer\n\nCode for the attached .jar.\n\n---\n\nFrom: developer\n\nI have been running in cluster mode as a yarn client. Just tried local and it ran fine, so progress...\n\nCluster is running CDH-5.4.0-1.cdh5.4.0.\n\n\n", "target_key": "SPARK-11885"} {"cluster": "JIRA-SPARK-ab69a396fd9c", "issue_id": "12975443", "key": "SPARK-15755", "n_comments": 3, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: java.lang.NullPointerException when run spark 2.0 setting spark.serializer=org.apache.spark.serializer.KryoSerializer\n\njava.lang.NullPointerException when run spark 2.0 setting spark.serializer=org.apache.spark.serializer.KryoSerializer\n\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 INFO YarnScheduler: Cancelling stage 6\n16/05/27 15:15:28 INFO YarnScheduler: Stage 6 was cancelled\n16/05/27 15:15:28 INFO DAGScheduler: ResultStage 6 (run at AccessController.java:-2) failed in 1.075 s\n16/05/27 15:15:28 WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0\n16/05/27 15:15:28 INFO DAGScheduler: Job 4 failed: run at AccessController.java:-2, took 14.847507 s\n16/05/27 15:15:28 ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING, \norg.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1863)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1926)", "query_30": "From: reporter\nSubject: java.lang.NullPointerException when run spark 2.0 setting spark.serializer=org.apache.spark.serializer.KryoSerializer\n\njava.lang.NullPointerException when run spark 2.0 setting spark.serializer=org.apache.spark.serializer.KryoSerializer\n\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 INFO YarnScheduler: Cancelling stage 6\n16/05/27 15:15:28 INFO YarnScheduler: Stage 6 was cancelled\n16/05/27 15:15:28 INFO DAGScheduler: ResultStage 6 (run at AccessController.java:-2) failed in 1.075 s\n16/05/27 15:15:28 WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0\n16/05/27 15:15:28 INFO DAGScheduler: Job 4 failed: run at AccessController.java:-2, took 14.847507 s\n16/05/27 15:15:28 ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING, \norg.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1863)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1926)\n\n---\n\nFrom: developer\n\nAny comments?", "query_60": "From: reporter\nSubject: java.lang.NullPointerException when run spark 2.0 setting spark.serializer=org.apache.spark.serializer.KryoSerializer\n\njava.lang.NullPointerException when run spark 2.0 setting spark.serializer=org.apache.spark.serializer.KryoSerializer\n\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 ERROR TaskResultGetter: Exception while getting task result\ncom.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)\n\tat com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)\n\tat com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)\n\tat com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)\n\tat org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)\n\tat org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)\n\tat org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1793)\n\tat org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NullPointerException\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)\n\tat org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)\n\tat scala.math.Ordering$$anon$4.compare(Ordering.scala:111)\n\tat java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:649)\n\tat java.util.PriorityQueue.siftUp(PriorityQueue.java:627)\n\tat java.util.PriorityQueue.offer(PriorityQueue.java:329)\n\tat java.util.PriorityQueue.add(PriorityQueue.java:306)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)\n\tat com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)\n\tat com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711)\n\tat com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)\n\t... 15 more\n16/05/27 15:15:28 INFO YarnScheduler: Cancelling stage 6\n16/05/27 15:15:28 INFO YarnScheduler: Stage 6 was cancelled\n16/05/27 15:15:28 INFO DAGScheduler: ResultStage 6 (run at AccessController.java:-2) failed in 1.075 s\n16/05/27 15:15:28 WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0\n16/05/27 15:15:28 INFO DAGScheduler: Job 4 failed: run at AccessController.java:-2, took 14.847507 s\n16/05/27 15:15:28 ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING, \norg.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException\nSerialization trace:\nunderlying (org.apache.spark.util.BoundedPriorityQueue)\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)\n\tat scala.Option.foreach(Option.scala:257)\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1863)\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:1926)\n\n---\n\nFrom: developer\n\nAny comments?\n\n---\n\nFrom: developer\n\nCould you provide a test case to reproduce the issue?", "target_key": "SPARK-14752"} {"cluster": "JIRA-SPARK-b89aff5afa41", "issue_id": "12981220", "key": "SPARK-16091", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: Dataset.partitionBy.csv raise a java.io.FileNotFoundException when launched on an hadoop cluster\n\nWhen writing a Dataset in a CSV file, the following exception java.io.FileNotFoundException is raised *after* the writing is done and successful.\n\nThis behaviour does not happen when the spark application is launched locally ; it should be related to hdfs management.\n\nHere is a test code:\n{code}\nimport org.apache.spark.SparkContext\nimport org.apache.hadoop.fs.FileSystem\nimport org.apache.hadoop.fs.{Path, PathFilter}\nimport org.apache.spark.sql.SQLContext\n\n\n\ncase class Test(A: String, B: String, C:String){\n}\n\nobject WriteTest {\n val sc: SparkContext = new SparkContext()\n val fs: FileSystem = FileSystem.get(sc.hadoopConfiguration)\n val sqlContext: SQLContext = new SQLContext(sc)\n \n import sqlContext.implicits._\n\n def main(args: Array[String]):Unit = {\n\n val ds = Seq(\n Test(\"abc\", \"abc\", \"abc\"),\n Test(\"abc\", \"abc\", \"def\"), \n Test(\"abc\", \"abc\", \"ghi\"), \n Test(\"abc\", \"xyz\", \"abc\"),\n Test(\"xyz\", \"xyz\", \"abc\")\n ).toDS()\n\n\n // works\n ds\n .write\n .option(\"header\",true)\n .mode(\"overwrite\")\n .csv(\"/tmp/test1.csv\")\n\n // fails\n ds\n .write\n .option(\"header\",true)\n .mode(\"overwrite\")\n .partitionBy(\"A\", \"B\")\n .csv(\"/tmp/test2.csv\")\n\n }\n}\n{code}\n\nand here is the exception stack:\n{code}\njava.io.FileNotFoundException: Path is not a file: /tmp/test2.csv/A=abc\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:68)\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:54)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1795)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1738)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1718)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1690)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:519)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:337)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)\n\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:408)\n\tat org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)\n\tat org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)\n\tat org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)\n\tat org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)\n\tat org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:1260)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:220)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:216)\n\tat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:216)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:208)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1$$anonfun$apply$2.apply(ListingFileCatalog.scala:104)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1$$anonfun$apply$2.apply(ListingFileCatalog.scala:92)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1.apply(ListingFileCatalog.scala:92)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1.apply(ListingFileCatalog.scala:80)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)\n\tat scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.listLeafFiles(ListingFileCatalog.scala:80)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.refresh(ListingFileCatalog.scala:69)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.(ListingFileCatalog.scala:50)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:307)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:424)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:252)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:234)\n\tat org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:697)\n\tat WriteTest$.main(write.scala:42)\n\tat WriteTest.main(write.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:483)\n\tat org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:610)\n{code}\n\nAnd here is the result of hdfs dfs -ls /tmp/test2.csv:\n\n{code}\nFound 3 items\ndrwxr-xr-x - hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/A=abc\ndrwxr-xr-x - hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/A=xyz\n-rw-r--r-- 3 hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/_SUCCESS\n\n{code}\n\nI have no idea if the bug comes from the hdfs implementation or the way spark uses it.", "query_30": "From: reporter\nSubject: Dataset.partitionBy.csv raise a java.io.FileNotFoundException when launched on an hadoop cluster\n\nWhen writing a Dataset in a CSV file, the following exception java.io.FileNotFoundException is raised *after* the writing is done and successful.\n\nThis behaviour does not happen when the spark application is launched locally ; it should be related to hdfs management.\n\nHere is a test code:\n{code}\nimport org.apache.spark.SparkContext\nimport org.apache.hadoop.fs.FileSystem\nimport org.apache.hadoop.fs.{Path, PathFilter}\nimport org.apache.spark.sql.SQLContext\n\n\n\ncase class Test(A: String, B: String, C:String){\n}\n\nobject WriteTest {\n val sc: SparkContext = new SparkContext()\n val fs: FileSystem = FileSystem.get(sc.hadoopConfiguration)\n val sqlContext: SQLContext = new SQLContext(sc)\n \n import sqlContext.implicits._\n\n def main(args: Array[String]):Unit = {\n\n val ds = Seq(\n Test(\"abc\", \"abc\", \"abc\"),\n Test(\"abc\", \"abc\", \"def\"), \n Test(\"abc\", \"abc\", \"ghi\"), \n Test(\"abc\", \"xyz\", \"abc\"),\n Test(\"xyz\", \"xyz\", \"abc\")\n ).toDS()\n\n\n // works\n ds\n .write\n .option(\"header\",true)\n .mode(\"overwrite\")\n .csv(\"/tmp/test1.csv\")\n\n // fails\n ds\n .write\n .option(\"header\",true)\n .mode(\"overwrite\")\n .partitionBy(\"A\", \"B\")\n .csv(\"/tmp/test2.csv\")\n\n }\n}\n{code}\n\nand here is the exception stack:\n{code}\njava.io.FileNotFoundException: Path is not a file: /tmp/test2.csv/A=abc\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:68)\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:54)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1795)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1738)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1718)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1690)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:519)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:337)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)\n\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:408)\n\tat org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)\n\tat org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)\n\tat org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)\n\tat org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)\n\tat org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:1260)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:220)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:216)\n\tat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:216)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:208)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1$$anonfun$apply$2.apply(ListingFileCatalog.scala:104)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1$$anonfun$apply$2.apply(ListingFileCatalog.scala:92)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1.apply(ListingFileCatalog.scala:92)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1.apply(ListingFileCatalog.scala:80)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)\n\tat scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.listLeafFiles(ListingFileCatalog.scala:80)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.refresh(ListingFileCatalog.scala:69)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.(ListingFileCatalog.scala:50)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:307)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:424)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:252)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:234)\n\tat org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:697)\n\tat WriteTest$.main(write.scala:42)\n\tat WriteTest.main(write.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:483)\n\tat org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:610)\n{code}\n\nAnd here is the result of hdfs dfs -ls /tmp/test2.csv:\n\n{code}\nFound 3 items\ndrwxr-xr-x - hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/A=abc\ndrwxr-xr-x - hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/A=xyz\n-rw-r--r-- 3 hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/_SUCCESS\n\n{code}\n\nI have no idea if the bug comes from the hdfs implementation or the way spark uses it.\n\n---\n\nFrom: developer\n\n[~rgiot] have a look at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark first. Don't set Blocker, but this doesn't seem nearly that important.\n\nYou need to write an hdfs:// URI if you intend to reference an HDFS path.", "query_60": "From: reporter\nSubject: Dataset.partitionBy.csv raise a java.io.FileNotFoundException when launched on an hadoop cluster\n\nWhen writing a Dataset in a CSV file, the following exception java.io.FileNotFoundException is raised *after* the writing is done and successful.\n\nThis behaviour does not happen when the spark application is launched locally ; it should be related to hdfs management.\n\nHere is a test code:\n{code}\nimport org.apache.spark.SparkContext\nimport org.apache.hadoop.fs.FileSystem\nimport org.apache.hadoop.fs.{Path, PathFilter}\nimport org.apache.spark.sql.SQLContext\n\n\n\ncase class Test(A: String, B: String, C:String){\n}\n\nobject WriteTest {\n val sc: SparkContext = new SparkContext()\n val fs: FileSystem = FileSystem.get(sc.hadoopConfiguration)\n val sqlContext: SQLContext = new SQLContext(sc)\n \n import sqlContext.implicits._\n\n def main(args: Array[String]):Unit = {\n\n val ds = Seq(\n Test(\"abc\", \"abc\", \"abc\"),\n Test(\"abc\", \"abc\", \"def\"), \n Test(\"abc\", \"abc\", \"ghi\"), \n Test(\"abc\", \"xyz\", \"abc\"),\n Test(\"xyz\", \"xyz\", \"abc\")\n ).toDS()\n\n\n // works\n ds\n .write\n .option(\"header\",true)\n .mode(\"overwrite\")\n .csv(\"/tmp/test1.csv\")\n\n // fails\n ds\n .write\n .option(\"header\",true)\n .mode(\"overwrite\")\n .partitionBy(\"A\", \"B\")\n .csv(\"/tmp/test2.csv\")\n\n }\n}\n{code}\n\nand here is the exception stack:\n{code}\njava.io.FileNotFoundException: Path is not a file: /tmp/test2.csv/A=abc\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:68)\n\tat org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:54)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1795)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1738)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1718)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1690)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:519)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:337)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)\n\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:408)\n\tat org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)\n\tat org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)\n\tat org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1222)\n\tat org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210)\n\tat org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:1260)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:220)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:216)\n\tat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:216)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:208)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1$$anonfun$apply$2.apply(ListingFileCatalog.scala:104)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1$$anonfun$apply$2.apply(ListingFileCatalog.scala:92)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:234)\n\tat scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1.apply(ListingFileCatalog.scala:92)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog$$anonfun$1.apply(ListingFileCatalog.scala:80)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n\tat scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)\n\tat scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)\n\tat scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.listLeafFiles(ListingFileCatalog.scala:80)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.refresh(ListingFileCatalog.scala:69)\n\tat org.apache.spark.sql.execution.datasources.ListingFileCatalog.(ListingFileCatalog.scala:50)\n\tat org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:307)\n\tat org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:424)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:252)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:234)\n\tat org.apache.spark.sql.DataFrameWriter.csv(DataFrameWriter.scala:697)\n\tat WriteTest$.main(write.scala:42)\n\tat WriteTest.main(write.scala)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:483)\n\tat org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:610)\n{code}\n\nAnd here is the result of hdfs dfs -ls /tmp/test2.csv:\n\n{code}\nFound 3 items\ndrwxr-xr-x - hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/A=abc\ndrwxr-xr-x - hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/A=xyz\n-rw-r--r-- 3 hadoop supergroup 0 2016-06-21 14:59 /tmp/test2.csv/_SUCCESS\n\n{code}\n\nI have no idea if the bug comes from the hdfs implementation or the way spark uses it.\n\n---\n\nFrom: developer\n\n[~rgiot] have a look at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark first. Don't set Blocker, but this doesn't seem nearly that important.\n\nYou need to write an hdfs:// URI if you intend to reference an HDFS path.\n\n---\n\nFrom: developer\n\nOk, sorry for the too high importance of the bug.\nNote that, on my configuration, the behaviour is exactly the same if I explicitly prepend the paths by hdfs://: ", "target_key": "SPARK-14959"} {"cluster": "JIRA-SPARK-7bd344e3c947", "issue_id": "12987868", "key": "SPARK-16449", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "SPARK", "query_0": "From: reporter\nSubject: unionAll raises \"Task not serializable\"\n\nGoal: Take the output from `describe` on a large DataFrame, then use a loop to calculate `skewness` and `kurtosis` from pyspark.sql.functions for each column, build them into a DataFrame of two rows, then use `unionAll` to merge them together.\n\nIssue: Despite having the same column names, in the same order with the same dtypes, the `unionAll` fails with \"Task not serializable\". However, if I build two test rows using dummy data then `unionAll` works fine. Also, if I collect my results then turn them straight back into DataFrames, `unionAll` succeeds. \n\n\nStep-by-step code and output with comments can be seen here: https://github.com/UrbanInstitute/pyspark-tutorials/blob/master/unionAll%20error.ipynb\nThe issue appears to be in the way the loop in code block 6 is building the rows before parallelizing, but the results look no different from the test rows that do work. I reproduced this on multiple datasets, so downloading the notebook and pointing it to any data of your own should replicate it.", "query_30": "From: reporter\nSubject: unionAll raises \"Task not serializable\"\n\nGoal: Take the output from `describe` on a large DataFrame, then use a loop to calculate `skewness` and `kurtosis` from pyspark.sql.functions for each column, build them into a DataFrame of two rows, then use `unionAll` to merge them together.\n\nIssue: Despite having the same column names, in the same order with the same dtypes, the `unionAll` fails with \"Task not serializable\". However, if I build two test rows using dummy data then `unionAll` works fine. Also, if I collect my results then turn them straight back into DataFrames, `unionAll` succeeds. \n\n\nStep-by-step code and output with comments can be seen here: https://github.com/UrbanInstitute/pyspark-tutorials/blob/master/unionAll%20error.ipynb\nThe issue appears to be in the way the loop in code block 6 is building the rows before parallelizing, but the results look no different from the test rows that do work. I reproduced this on multiple datasets, so downloading the notebook and pointing it to any data of your own should replicate it.\n\n---\n\nFrom: developer\n\nInteresting. Looks like something is serializing a Scala Iterator and it doesn't work. \n\nThe relevant subset is below. Hm, maybe something in LocalTableScan can be rejiggered to avoid this.\n\n{code}\nCaused by: java.io.NotSerializableException: scala.collection.Iterator$$anon$11\nSerialization stack:\n\t- object not serializable (class: scala.collection.Iterator$$anon$11, value: empty iterator)\n\t- field (class: scala.collection.Iterator$$anonfun$toStream$1, name: $outer, type: interface scala.collection.Iterator)\n\t- object (class scala.collection.Iterator$$anonfun$toStream$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream(WrappedArray(3526154, 3526154, 1580402, 3526154, 3526154), WrappedArray(5.50388599500189E11, 4.178168090221903, 234846.780654818, 5.134865351881966, 354.7084951479714), WrappedArray(2.596112361975223E11, 0.34382335723646484, 118170.68592261613, 3.3833930336063456, 4.011812510792076), WrappedArray(100002091588, 2.75, 0.85, -1, 292), WrappedArray(999995696635, 6.125, 1193544.39, 34, 480)))\n\t- field (class: scala.collection.immutable.Stream$$anonfun$zip$1, name: $outer, type: class scala.collection.immutable.Stream)\n\t- object (class scala.collection.immutable.Stream$$anonfun$zip$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream((WrappedArray(3526154, 3526154, 1580402, 3526154, 3526154),(count,)), (WrappedArray(5.50388599500189E11, 4.178168090221903, 234846.780654818, 5.134865351881966, 354.7084951479714),(mean,)), (WrappedArray(2.596112361975223E11, 0.34382335723646484, 118170.68592261613, 3.3833930336063456, 4.011812510792076),(stddev,)), (WrappedArray(100002091588, 2.75, 0.85, -1, 292),(min,)), (WrappedArray(999995696635, 6.125, 1193544.39, 34, 480),(max,))))\n\t- field (class: scala.collection.immutable.Stream$$anonfun$map$1, name: $outer, type: class scala.collection.immutable.Stream)\n\t- object (class scala.collection.immutable.Stream$$anonfun$map$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream([count,3526154,3526154,1580402,3526154,3526154], [mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714], [stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076], [min,100002091588,2.75,0.85,-1,292], [max,999995696635,6.125,1193544.39,34,480]))\n\t- field (class: scala.collection.immutable.Stream$$anonfun$map$1, name: $outer, type: class scala.collection.immutable.Stream)\n\t- object (class scala.collection.immutable.Stream$$anonfun$map$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream([count,3526154,3526154,1580402,3526154,3526154], [mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714], [stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076], [min,100002091588,2.75,0.85,-1,292], [max,999995696635,6.125,1193544.39,34,480]))\n\t- field (class: org.apache.spark.sql.execution.LocalTableScan, name: rows, type: interface scala.collection.Seq)\n\t- object (class org.apache.spark.sql.execution.LocalTableScan, LocalTableScan [summary#228,C0#229,C3#230,C4#231,C5#232,C6#233], [[count,3526154,3526154,1580402,3526154,3526154],[mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714],[stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076],[min,100002091588,2.75,0.85,-1,292],[max,999995696635,6.125,1193544.39,34,480]]\n)\n\t- field (class: org.apache.spark.sql.execution.ConvertToUnsafe, name: child, type: class org.apache.spark.sql.execution.SparkPlan)\n\t- object (class org.apache.spark.sql.execution.ConvertToUnsafe, ConvertToUnsafe\n+- LocalTableScan [summary#228,C0#229,C3#230,C4#231,C5#232,C6#233], [[count,3526154,3526154,1580402,3526154,3526154],[mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714],[stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076],[min,100002091588,2.75,0.85,-1,292],[max,999995696635,6.125,1193544.39,34,480]]\n)\n\t- field (class: org.apache.spark.sql.execution.ConvertToUnsafe$$anonfun$1, name: $outer, type: class org.apache.spark.sql.execution.ConvertToUnsafe)\n\t- object (class org.apache.spark.sql.execution.ConvertToUnsafe$$anonfun$1, )\n{code}", "query_60": "From: reporter\nSubject: unionAll raises \"Task not serializable\"\n\nGoal: Take the output from `describe` on a large DataFrame, then use a loop to calculate `skewness` and `kurtosis` from pyspark.sql.functions for each column, build them into a DataFrame of two rows, then use `unionAll` to merge them together.\n\nIssue: Despite having the same column names, in the same order with the same dtypes, the `unionAll` fails with \"Task not serializable\". However, if I build two test rows using dummy data then `unionAll` works fine. Also, if I collect my results then turn them straight back into DataFrames, `unionAll` succeeds. \n\n\nStep-by-step code and output with comments can be seen here: https://github.com/UrbanInstitute/pyspark-tutorials/blob/master/unionAll%20error.ipynb\nThe issue appears to be in the way the loop in code block 6 is building the rows before parallelizing, but the results look no different from the test rows that do work. I reproduced this on multiple datasets, so downloading the notebook and pointing it to any data of your own should replicate it.\n\n---\n\nFrom: developer\n\nInteresting. Looks like something is serializing a Scala Iterator and it doesn't work. \n\nThe relevant subset is below. Hm, maybe something in LocalTableScan can be rejiggered to avoid this.\n\n{code}\nCaused by: java.io.NotSerializableException: scala.collection.Iterator$$anon$11\nSerialization stack:\n\t- object not serializable (class: scala.collection.Iterator$$anon$11, value: empty iterator)\n\t- field (class: scala.collection.Iterator$$anonfun$toStream$1, name: $outer, type: interface scala.collection.Iterator)\n\t- object (class scala.collection.Iterator$$anonfun$toStream$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream(WrappedArray(3526154, 3526154, 1580402, 3526154, 3526154), WrappedArray(5.50388599500189E11, 4.178168090221903, 234846.780654818, 5.134865351881966, 354.7084951479714), WrappedArray(2.596112361975223E11, 0.34382335723646484, 118170.68592261613, 3.3833930336063456, 4.011812510792076), WrappedArray(100002091588, 2.75, 0.85, -1, 292), WrappedArray(999995696635, 6.125, 1193544.39, 34, 480)))\n\t- field (class: scala.collection.immutable.Stream$$anonfun$zip$1, name: $outer, type: class scala.collection.immutable.Stream)\n\t- object (class scala.collection.immutable.Stream$$anonfun$zip$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream((WrappedArray(3526154, 3526154, 1580402, 3526154, 3526154),(count,)), (WrappedArray(5.50388599500189E11, 4.178168090221903, 234846.780654818, 5.134865351881966, 354.7084951479714),(mean,)), (WrappedArray(2.596112361975223E11, 0.34382335723646484, 118170.68592261613, 3.3833930336063456, 4.011812510792076),(stddev,)), (WrappedArray(100002091588, 2.75, 0.85, -1, 292),(min,)), (WrappedArray(999995696635, 6.125, 1193544.39, 34, 480),(max,))))\n\t- field (class: scala.collection.immutable.Stream$$anonfun$map$1, name: $outer, type: class scala.collection.immutable.Stream)\n\t- object (class scala.collection.immutable.Stream$$anonfun$map$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream([count,3526154,3526154,1580402,3526154,3526154], [mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714], [stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076], [min,100002091588,2.75,0.85,-1,292], [max,999995696635,6.125,1193544.39,34,480]))\n\t- field (class: scala.collection.immutable.Stream$$anonfun$map$1, name: $outer, type: class scala.collection.immutable.Stream)\n\t- object (class scala.collection.immutable.Stream$$anonfun$map$1, )\n\t- field (class: scala.collection.immutable.Stream$Cons, name: tl, type: interface scala.Function0)\n\t- object (class scala.collection.immutable.Stream$Cons, Stream([count,3526154,3526154,1580402,3526154,3526154], [mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714], [stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076], [min,100002091588,2.75,0.85,-1,292], [max,999995696635,6.125,1193544.39,34,480]))\n\t- field (class: org.apache.spark.sql.execution.LocalTableScan, name: rows, type: interface scala.collection.Seq)\n\t- object (class org.apache.spark.sql.execution.LocalTableScan, LocalTableScan [summary#228,C0#229,C3#230,C4#231,C5#232,C6#233], [[count,3526154,3526154,1580402,3526154,3526154],[mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714],[stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076],[min,100002091588,2.75,0.85,-1,292],[max,999995696635,6.125,1193544.39,34,480]]\n)\n\t- field (class: org.apache.spark.sql.execution.ConvertToUnsafe, name: child, type: class org.apache.spark.sql.execution.SparkPlan)\n\t- object (class org.apache.spark.sql.execution.ConvertToUnsafe, ConvertToUnsafe\n+- LocalTableScan [summary#228,C0#229,C3#230,C4#231,C5#232,C6#233], [[count,3526154,3526154,1580402,3526154,3526154],[mean,5.50388599500189E11,4.178168090221903,234846.780654818,5.134865351881966,354.7084951479714],[stddev,2.596112361975223E11,0.34382335723646484,118170.68592261613,3.3833930336063456,4.011812510792076],[min,100002091588,2.75,0.85,-1,292],[max,999995696635,6.125,1193544.39,34,480]]\n)\n\t- field (class: org.apache.spark.sql.execution.ConvertToUnsafe$$anonfun$1, name: $outer, type: class org.apache.spark.sql.execution.ConvertToUnsafe)\n\t- object (class org.apache.spark.sql.execution.ConvertToUnsafe$$anonfun$1, )\n{code}", "target_key": "SPARK-16173"} {"cluster": "JIRA-SPARK-dcb981be150c", "issue_id": "13065031", "key": "SPARK-20387", "n_comments": 4, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: Permissive mode is not replacing corrupt record with null\n\nWhen reading the below mentioned time value by specifying \"mode\" as PERMISSIVE.\n\nSource File: \nString,int,f1,bool1\nabc,23111,23.07738,true\nabc,23111,23.07738,true\nabc,23111,true,true\n\nSource code1:\nDataset dataset = getSqlContext().read()\n.option(PARSER_LIB, \"commons\")\n.option(INFER_SCHEMA, \"true\")\n.option(DELIMITER, \",\")\n.option(QUOTE, \"\\\"\")\n.option(MODE, Mode.PERMISSIVE)\n.csv(sourceFile);\ndataset.collect();\n\nResult: Error is thrown\nstack trace: \nERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)\njava.lang.IllegalArgumentException: For input string: \"23.07738\"\n at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290)\n at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260)\n at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:29)\n at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:270)\n at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:125)\n at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:94)\n at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:167)\n at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:166)", "query_30": "From: reporter\nSubject: Permissive mode is not replacing corrupt record with null\n\nWhen reading the below mentioned time value by specifying \"mode\" as PERMISSIVE.\n\nSource File: \nString,int,f1,bool1\nabc,23111,23.07738,true\nabc,23111,23.07738,true\nabc,23111,true,true\n\nSource code1:\nDataset dataset = getSqlContext().read()\n.option(PARSER_LIB, \"commons\")\n.option(INFER_SCHEMA, \"true\")\n.option(DELIMITER, \",\")\n.option(QUOTE, \"\\\"\")\n.option(MODE, Mode.PERMISSIVE)\n.csv(sourceFile);\ndataset.collect();\n\nResult: Error is thrown\nstack trace: \nERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)\njava.lang.IllegalArgumentException: For input string: \"23.07738\"\n at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290)\n at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260)\n at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:29)\n at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:270)\n at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:125)\n at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:94)\n at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:167)\n at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:166)\n\n---\n\nFrom: developer\n\nI am pretty sure that this is fixed in 2.2.0 in that JIRA. I am resolving this. \n\n{code}\n val ds = Seq(\"String,int,f1,bool1\",\n \"abc,23111,23.07738,true\",\n \"abc,23111,23.07738,true\",\n \"abc,23111,true,true\").toDS()\n\n spark.read\n .option(\"sep\", \",\")\n .option(\"quote\", \"\\\"\")\n .option(\"header\", true)\n .option(\"inferSchema\", true).csv(ds).show()\n{code}\n\nprints\n\n{code}\n+------+-----+--------+-----+\n|String| int| f1|bool1|\n+------+-----+--------+-----+\n| abc|23111|23.07738| true|\n| abc|23111|23.07738| true|\n| abc|23111| true| true|\n+------+-----+--------+-----+\n{code}\n\n\n---\n\nFrom: developer\n\nNot all the corrupted values are replacing with null. Refer the below-given scenario:\r\n\r\nSource File: \r\n'Col1','Col2','Col3','Col4','Col5','Col6',\r\n'1000','abc','10yui000','400','20.8','2003-03-04',\r\n'1001','xyz','30000','4000','20.8','2003-03-04',\r\n'1002','abc','40000','40,000','20.8','2003-03-04'\r\n'1003','xyz','50000','40,0000','20.8','2003-03-04'\r\n'1004','abc','60000','40,000','20.8','2003-03-04'\r\n\r\nUser_defined_Schema:\r\n[{\r\n\t\t\t\"dataType\": \"integer\",\r\n\t\t\t\"type\": \"Measure\",\r\n\t\t\t\"name\": \"Col1\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"string\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"Col2\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"float\",\r\n\t\t\t\"type\": \"Measure\",\r\n\t\t\t\"name\": \"Col3\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"string\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"Col4\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"double\",\r\n\t\t\t\"type\": \"Measure\",\r\n\t\t\t\"name\": \"Col5\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"date\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"Col6\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"string\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"_c6\"\r\n\t\t}\r\n\r\nSource code1:\r\nDataset dataset =sparkSession.read().schema(User_defined_Schema)\r\n.option(PARSER_LIB, \"commons\")\r\n.option(DELIMITER, \",\")\r\n.option(QUOTE, \"\\\"\")\r\n.option(MODE, Mode.PERMISSIVE)\r\n.csv(sourceFile);\r\n\r\ndataset.collect();\r\nResult: 10yui000 is parsed as 10\r\nRow : '1000','abc','10','400','20.8','2003-03-04',\r\n\r\nExpected: According to the PERMISSIVE mode, 10yui000 should be replaced with null.\r\n", "query_60": "From: reporter\nSubject: Permissive mode is not replacing corrupt record with null\n\nWhen reading the below mentioned time value by specifying \"mode\" as PERMISSIVE.\n\nSource File: \nString,int,f1,bool1\nabc,23111,23.07738,true\nabc,23111,23.07738,true\nabc,23111,true,true\n\nSource code1:\nDataset dataset = getSqlContext().read()\n.option(PARSER_LIB, \"commons\")\n.option(INFER_SCHEMA, \"true\")\n.option(DELIMITER, \",\")\n.option(QUOTE, \"\\\"\")\n.option(MODE, Mode.PERMISSIVE)\n.csv(sourceFile);\ndataset.collect();\n\nResult: Error is thrown\nstack trace: \nERROR Executor: Exception in task 0.0 in stage 15.0 (TID 15)\njava.lang.IllegalArgumentException: For input string: \"23.07738\"\n at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290)\n at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260)\n at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:29)\n at org.apache.spark.sql.execution.datasources.csv.CSVTypeCast$.castTo(CSVInferSchema.scala:270)\n at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:125)\n at org.apache.spark.sql.execution.datasources.csv.CSVRelation$$anonfun$csvParser$3.apply(CSVRelation.scala:94)\n at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:167)\n at org.apache.spark.sql.execution.datasources.csv.CSVFileFormat$$anonfun$buildReader$1$$anonfun$apply$2.apply(CSVFileFormat.scala:166)\n\n---\n\nFrom: developer\n\nI am pretty sure that this is fixed in 2.2.0 in that JIRA. I am resolving this. \n\n{code}\n val ds = Seq(\"String,int,f1,bool1\",\n \"abc,23111,23.07738,true\",\n \"abc,23111,23.07738,true\",\n \"abc,23111,true,true\").toDS()\n\n spark.read\n .option(\"sep\", \",\")\n .option(\"quote\", \"\\\"\")\n .option(\"header\", true)\n .option(\"inferSchema\", true).csv(ds).show()\n{code}\n\nprints\n\n{code}\n+------+-----+--------+-----+\n|String| int| f1|bool1|\n+------+-----+--------+-----+\n| abc|23111|23.07738| true|\n| abc|23111|23.07738| true|\n| abc|23111| true| true|\n+------+-----+--------+-----+\n{code}\n\n\n---\n\nFrom: developer\n\nNot all the corrupted values are replacing with null. Refer the below-given scenario:\r\n\r\nSource File: \r\n'Col1','Col2','Col3','Col4','Col5','Col6',\r\n'1000','abc','10yui000','400','20.8','2003-03-04',\r\n'1001','xyz','30000','4000','20.8','2003-03-04',\r\n'1002','abc','40000','40,000','20.8','2003-03-04'\r\n'1003','xyz','50000','40,0000','20.8','2003-03-04'\r\n'1004','abc','60000','40,000','20.8','2003-03-04'\r\n\r\nUser_defined_Schema:\r\n[{\r\n\t\t\t\"dataType\": \"integer\",\r\n\t\t\t\"type\": \"Measure\",\r\n\t\t\t\"name\": \"Col1\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"string\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"Col2\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"float\",\r\n\t\t\t\"type\": \"Measure\",\r\n\t\t\t\"name\": \"Col3\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"string\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"Col4\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"double\",\r\n\t\t\t\"type\": \"Measure\",\r\n\t\t\t\"name\": \"Col5\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"date\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"Col6\"\r\n\t\t},\r\n\t\t{\r\n\t\t\t\"dataType\": \"string\",\r\n\t\t\t\"type\": \"Dimension\",\r\n\t\t\t\"name\": \"_c6\"\r\n\t\t}\r\n\r\nSource code1:\r\nDataset dataset =sparkSession.read().schema(User_defined_Schema)\r\n.option(PARSER_LIB, \"commons\")\r\n.option(DELIMITER, \",\")\r\n.option(QUOTE, \"\\\"\")\r\n.option(MODE, Mode.PERMISSIVE)\r\n.csv(sourceFile);\r\n\r\ndataset.collect();\r\nResult: 10yui000 is parsed as 10\r\nRow : '1000','abc','10','400','20.8','2003-03-04',\r\n\r\nExpected: According to the PERMISSIVE mode, 10yui000 should be replaced with null.\r\n\n\n---\n\nFrom: developer\n\nThat's not the same example. I believe the underlying number parsing from the JDK will parse, in permissive move, anything it can as a string as a number and ignore the rest. I think that's consistent then. [~hyukjin.kwon]?\r\nThere also appears to be a problem with your input -- extra blank column.", "target_key": "SPARK-18699"} {"cluster": "JIRA-SPARK-fdca525f0b01", "issue_id": "13067172", "key": "SPARK-20480", "n_comments": 2, "progress_valid": {"0": true, "30": true, "60": true}, "project": "SPARK", "query_0": "From: reporter\nSubject: FileFormatWriter hides FetchFailedException from scheduler\n\nI was running a large job where it was getting faiures, noticed they were listed as \"SparkException: Task failed while writing rows\", but when I looked further they were really caused by FetchFailure exceptions. This is a problem because the scheduler handles Fetch Failures differently then normal exception. This can affect things like blacklisting.\n\n\n\n{noformat}\n17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID 102902)\norg.apache.spark.SparkException: Task failed while writing rows\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:99)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect to host1.com:7337\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\n\t... 8 more\nCaused by: java.io.IOException: Failed to connect to host1.com:7337\n\tat org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)\n\tat org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)\n\tat org.apache.spark.network.shuffle.ExternalShuffleClient$1.createAndStart(ExternalShuffleClient.java:105)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)\n\t... 1 more\nCaused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host1.com:7337\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\tat io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)\n\t... 2 more\n{noformat}\n", "query_30": "From: reporter\nSubject: FileFormatWriter hides FetchFailedException from scheduler\n\nI was running a large job where it was getting faiures, noticed they were listed as \"SparkException: Task failed while writing rows\", but when I looked further they were really caused by FetchFailure exceptions. This is a problem because the scheduler handles Fetch Failures differently then normal exception. This can affect things like blacklisting.\n\n\n\n{noformat}\n17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID 102902)\norg.apache.spark.SparkException: Task failed while writing rows\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:99)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect to host1.com:7337\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\n\t... 8 more\nCaused by: java.io.IOException: Failed to connect to host1.com:7337\n\tat org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)\n\tat org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)\n\tat org.apache.spark.network.shuffle.ExternalShuffleClient$1.createAndStart(ExternalShuffleClient.java:105)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)\n\t... 1 more\nCaused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host1.com:7337\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\tat io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)\n\t... 2 more\n{noformat}\n\n\n---\n\nFrom: developer\n\nexception in task manager looks like:\n17/04/26 20:09:21 INFO TaskSetManager: Lost task 3516.0 in stage 4.0 (TID 103691) on gsbl521n33.blue.ygrid.yahoo.com, exec\nutor 4516: org.apache.spark.SparkException (Task failed while writing rows) [duplicate 22]", "query_60": "From: reporter\nSubject: FileFormatWriter hides FetchFailedException from scheduler\n\nI was running a large job where it was getting faiures, noticed they were listed as \"SparkException: Task failed while writing rows\", but when I looked further they were really caused by FetchFailure exceptions. This is a problem because the scheduler handles Fetch Failures differently then normal exception. This can affect things like blacklisting.\n\n\n\n{noformat}\n17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID 102902)\norg.apache.spark.SparkException: Task failed while writing rows\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)\n\tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\n\tat org.apache.spark.scheduler.Task.run(Task.scala:99)\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect to host1.com:7337\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)\n\tat org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:409)\n\tat scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)\n\tat scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)\n\tat org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)\n\tat scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown Source)\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)\n\tat org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)\n\tat org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)\n\tat org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)\n\t... 8 more\nCaused by: java.io.IOException: Failed to connect to host1.com:7337\n\tat org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)\n\tat org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)\n\tat org.apache.spark.network.shuffle.ExternalShuffleClient$1.createAndStart(ExternalShuffleClient.java:105)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)\n\tat org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)\n\t... 1 more\nCaused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host1.com:7337\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)\n\tat io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:291)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)\n\t... 2 more\n{noformat}\n\n\n---\n\nFrom: developer\n\nexception in task manager looks like:\n17/04/26 20:09:21 INFO TaskSetManager: Lost task 3516.0 in stage 4.0 (TID 103691) on gsbl521n33.blue.ygrid.yahoo.com, exec\nutor 4516: org.apache.spark.SparkException (Task failed while writing rows) [duplicate 22]\n\n---\n\nFrom: developer\n\nNote with blacklisting on this caused the job to fail:\nJob aborted due to stage failure: Aborting TaskSet 4.0 because task 21 (partition 21) cannot run anywhere due to node and executor blacklist. Blacklisting behavior can be configured via spark.blacklist.*.", "target_key": "SPARK-19276"} {"cluster": "JIRA-SPARK-aeb0851a7b1c", "issue_id": "13118184", "key": "SPARK-22517", "n_comments": 1, "progress_valid": {"0": true, "30": true, "60": false}, "project": "SPARK", "query_0": "From: reporter\nSubject: NullPointerException in ShuffleExternalSorter.spill()\n\nI see a NullPointerException during sorting with the following stacktrace:\r\n{code}\r\n17/11/13 15:02:56 ERROR Executor: Exception in task 138.0 in stage 9.0 (TID 13497)\r\njava.lang.NullPointerException\r\n at org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:383)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:193)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.spill(ShuffleExternalSorter.java:254)\r\n at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:203)\r\n at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:281)\r\n at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:90)\r\n at org.apache.spark.shuffle.sort.ShuffleInMemorySorter.reset(ShuffleInMemorySorter.java:100)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.spill(ShuffleExternalSorter.java:256)\r\n at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:203)\r\n at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:281)\r\n at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:90)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.growPointerArrayIfNecessary(ShuffleExternalSorter.java:328)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:379)\r\n at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246)\r\n at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167)\r\n at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\r\n at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\r\n at org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)\r\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n at java.lang.Thread.run(Thread.java:748)\r\n{code}\r\n", "query_30": "From: reporter\nSubject: NullPointerException in ShuffleExternalSorter.spill()\n\nI see a NullPointerException during sorting with the following stacktrace:\r\n{code}\r\n17/11/13 15:02:56 ERROR Executor: Exception in task 138.0 in stage 9.0 (TID 13497)\r\njava.lang.NullPointerException\r\n at org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:383)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:193)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.spill(ShuffleExternalSorter.java:254)\r\n at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:203)\r\n at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:281)\r\n at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:90)\r\n at org.apache.spark.shuffle.sort.ShuffleInMemorySorter.reset(ShuffleInMemorySorter.java:100)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.spill(ShuffleExternalSorter.java:256)\r\n at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:203)\r\n at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:281)\r\n at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:90)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.growPointerArrayIfNecessary(ShuffleExternalSorter.java:328)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:379)\r\n at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246)\r\n at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167)\r\n at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\r\n at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\r\n at org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)\r\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n at java.lang.Thread.run(Thread.java:748)\r\n{code}\r\n\n\n---\n\nFrom: developer\n\nCould you please post a program that can reproduce this program?", "query_60": "From: reporter\nSubject: NullPointerException in ShuffleExternalSorter.spill()\n\nI see a NullPointerException during sorting with the following stacktrace:\r\n{code}\r\n17/11/13 15:02:56 ERROR Executor: Exception in task 138.0 in stage 9.0 (TID 13497)\r\njava.lang.NullPointerException\r\n at org.apache.spark.memory.TaskMemoryManager.getPage(TaskMemoryManager.java:383)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.writeSortedFile(ShuffleExternalSorter.java:193)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.spill(ShuffleExternalSorter.java:254)\r\n at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:203)\r\n at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:281)\r\n at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:90)\r\n at org.apache.spark.shuffle.sort.ShuffleInMemorySorter.reset(ShuffleInMemorySorter.java:100)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.spill(ShuffleExternalSorter.java:256)\r\n at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:203)\r\n at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:281)\r\n at org.apache.spark.memory.MemoryConsumer.allocateArray(MemoryConsumer.java:90)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.growPointerArrayIfNecessary(ShuffleExternalSorter.java:328)\r\n at org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:379)\r\n at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246)\r\n at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167)\r\n at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)\r\n at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)\r\n at org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)\r\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n at java.lang.Thread.run(Thread.java:748)\r\n{code}\r\n\n\n---\n\nFrom: developer\n\nCould you please post a program that can reproduce this program?", "target_key": "SPARK-21907"}