Мобильная версияЕсть четыре докера Bitnami. Один из них строится на локальной машине, а отдых строится на удаленных виртуальных машинах с хостом 11.119.99.11.119.99.11.119.9.3.3 соответственно. Одним из них с хозяином 11.119.9.1 и портом 7077 является узлом Spark-Master. Локальная машина подключается к удаленному через vpn. с режимом кластера. После выполнения Spark-Sumbit я проверил журнал Master Node of Spark Web UI и обнаружил ошибку " java.io.ioexception: не удалось подключиться к Spark-Master/11.119.8.1: 35369 " ниже. Кажется, что работник не смог подключиться к мастеру. < /P>
Код: Выделить всё
Spark Executor Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*"
"-Xmx1024M" "-Dspark.driver.port=35369" "-Dspark.cassandra.connection.port=9042"
"-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions"
"--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED"
"--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED"
"--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED"
"--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED"
"--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED"
"--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED"
"--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED"
"-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend"
"--driver-url" "spark://CoarseGrainedScheduler@spark-master:35369" "--executor-id" "17" "--hostname" "11.119.8.3"
"--cores" "4" "--app-id" "app-20250221160412-0007" "--worker-url" "spark://Worker@11.119.8.3:40277" "--resourceProfileId" "0"
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/02/21 16:04:26 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 51665@s-datacenter03
25/02/21 16:04:26 INFO SignalUtils: Registering signal handler for TERM
25/02/21 16:04:26 INFO SignalUtils: Registering signal handler for HUP
25/02/21 16:04:26 INFO SignalUtils: Registering signal handler for INT
25/02/21 16:04:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/02/21 16:04:26 INFO SecurityManager: Changing view acls to: spark
25/02/21 16:04:26 INFO SecurityManager: Changing modify acls to: spark
25/02/21 16:04:26 INFO SecurityManager: Changing view acls groups to:
25/02/21 16:04:26 INFO SecurityManager: Changing modify acls groups to:
25/02/21 16:04:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark;
groups with view permissions: EMPTY;
users with modify permissions: spark; groups with modify permissions: EMPTY
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:429)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:418)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:56)
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:310)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:449)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:447)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 4 more
Caused by: java.io.IOException: Failed to connect to spark-master/11.119.8.1:35369
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:294)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:226)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: spark-master/11.119.8.1:35369
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
< /code>
Верхняя часть моей головы, я, хотя это может быть проблемой сети, поэтому я проверяю сетевое соединение между удаленным мастер -виртуальной машиной и моим локальным докером. Это показывает, что сеть подключена. < /P>
root@spark-master:/opt/spark# telnet 11.119.8.1 7077
Trying 11.119.8.1...
Connected to 11.119.8.1.
Escape character is '^]'.
< /code>
Затем я проверяю соединение между главным узлом и рабочими узлами. Кажется, что соединения между узлами нормальны. < /P>
kaikaiwan@s-datacenter02:~$ telnet 11.119.8.1 7077
Trying 11.119.8.1...
Connected to 11.119.8.1.
Escape character is '^]'.
kaikaiwan@s-datacenter03:~$ telnet 11.119.8.1 7077
Trying 11.119.8.1...
Connected to 11.119.8.1.
Escape character is '^]'.
kaikaiwan@s-datacenter01:/$ telnet 11.119.8.2 7000
Trying 11.119.8.2...
Connected to 11.119.8.2.
< /code>
Тогда мой колледж запускает Spark-Submit непосредственно в Docker Spark Master на удаленном сервере. В журнале показывается «[b] 25/02/20 19:56:39 Info TransportClientFactory: успешно создано соединение с /11.119.8.1:34277 после 77 мс (0 мс потрачена в начальных началах) [/b]». Это, кажется, успешно создает связь между мастером и работником. < /P>
Spark Executor Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=34277" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@11.119.8.1:34277" "--executor-id" "1" "--hostname" "11.119.8.3" "--cores" "4" "--app-id" "app-20250220195646-0000" "--worker-url" "spark://Worker@11.119.8.3:40277" "--resourceProfileId" "0"
========================================
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/02/20 19:56:38 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 105@s-datacenter03
25/02/20 19:56:38 INFO SignalUtils: Registering signal handler for TERM
25/02/20 19:56:38 INFO SignalUtils: Registering signal handler for HUP
25/02/20 19:56:38 INFO SignalUtils: Registering signal handler for INT
25/02/20 19:56:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/02/20 19:56:38 INFO SecurityManager: Changing view acls to: spark
25/02/20 19:56:38 INFO SecurityManager: Changing modify acls to: spark
25/02/20 19:56:38 INFO SecurityManager: Changing view acls groups to:
25/02/20 19:56:38 INFO SecurityManager: Changing modify acls groups to:
25/02/20 19:56:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 77 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO SecurityManager: Changing view acls to: spark
25/02/20 19:56:39 INFO SecurityManager: Changing modify acls to: spark
25/02/20 19:56:39 INFO SecurityManager: Changing view acls groups to:
25/02/20 19:56:39 INFO SecurityManager: Changing modify acls groups to:
25/02/20 19:56:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 5 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO DiskBlockManager: Created local directory at /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/blockmgr-db0ccc37-43db-47ab-a2c3-fdaccc28b467
25/02/20 19:56:39 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@11.119.8.1:34277
25/02/20 19:56:39 INFO WorkerWatcher: Connecting to worker spark://Worker@11.119.8.3:40277
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.3:40277 after 9 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO WorkerWatcher: Successfully connected to spark://Worker@11.119.8.3:40277
25/02/20 19:56:39 INFO ResourceUtils: ==============================================================
25/02/20 19:56:39 INFO ResourceUtils: No custom resources configured for spark.executor.
25/02/20 19:56:39 INFO ResourceUtils: ==============================================================
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
25/02/20 19:56:39 INFO Executor: Starting executor ID 1 on host 11.110.9.3
25/02/20 19:56:39 INFO Executor: OS info Linux, 6.8.0-41-generic, amd64
25/02/20 19:56:39 INFO Executor: Java version 17.0.14
25/02/20 19:56:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37575.
25/02/20 19:56:39 INFO NettyBlockTransferService: Server created on 11.110.9.3:37575
25/02/20 19:56:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
25/02/20 19:56:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, 11.119.8.3, 37575, None)
25/02/20 19:56:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, 11.119.8.3, 37575, None)
25/02/20 19:56:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, 11.119.8.3, 37575, None)
25/02/20 19:56:39 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
25/02/20 19:56:39 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@61224013 for default.
25/02/20 19:56:39 INFO Executor: Fetching spark://11.119.8.1:34277/jars/spark-examples_2.12-3.5.4.jar with timestamp 1740052605090
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 2 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO Utils: Fetching spark://11.119.8.1:34277/jars/spark-examples_2.12-3.5.4.jar to /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7/fetchFileTemp9485593455226099991.tmp
25/02/20 19:56:39 INFO Utils: Copying /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7/15260078331740052605090_cache to /opt/bitnami/spark/work/app-20250220195646-0000/1/./spark-examples_2.12-3.5.4.jar
25/02/20 19:56:39 INFO Executor: Adding file:/opt/bitnami/spark/work/app-20250220195646-0000/1/./spark-examples_2.12-3.5.4.jar to class loader default
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 4
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 5
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 6
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 7
25/02/20 19:56:39 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)
25/02/20 19:56:39 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
25/02/20 19:56:39 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
25/02/20 19:56:39 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
25/02/20 19:56:40 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
25/02/20 19:56:40 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:36735 after 3 ms (0 ms spent in bootstraps)
25/02/20 19:56:40 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.3 KiB, free 434.4 MiB)
25/02/20 19:56:40 INFO TorrentBroadcast: Reading broadcast variable 0 took 117 ms
25/02/20 19:56:40 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.0 KiB, free 434.4 MiB)
25/02/20 19:56:40 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 8
25/02/20 19:56:40 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 9
25/02/20 19:56:40 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 10
25/02/20 19:56:40 INFO Executor: Running task 10.0 in stage 0.0 (TID 10)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 11
25/02/20 19:56:40 INFO Executor: Running task 11.0 in stage 0.0 (TID 11)
25/02/20 19:56:40 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 969 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 16
25/02/20 19:56:40 INFO Executor: Running task 16.0 in stage 0.0 (TID 16)
25/02/20 19:56:40 INFO Executor: Finished task 10.0 in stage 0.0 (TID 10). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 19
25/02/20 19:56:40 INFO Executor: Running task 19.0 in stage 0.0 (TID 19)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 20
25/02/20 19:56:40 INFO Executor: Running task 20.0 in stage 0.0 (TID 20)
25/02/20 19:56:40 INFO Executor: Finished task 16.0 in stage 0.0 (TID 16). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 11.0 in stage 0.0 (TID 11). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 24
25/02/20 19:56:40 INFO Executor: Running task 24.0 in stage 0.0 (TID 24)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 26
25/02/20 19:56:40 INFO Executor: Running task 26.0 in stage 0.0 (TID 26)
25/02/20 19:56:40 INFO Executor: Finished task 19.0 in stage 0.0 (TID 19). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 20.0 in stage 0.0 (TID 20). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 28
25/02/20 19:56:40 INFO Executor: Running task 28.0 in stage 0.0 (TID 28)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 29
25/02/20 19:56:40 INFO Executor: Running task 29.0 in stage 0.0 (TID 29)
25/02/20 19:56:40 INFO Executor: Finished task 24.0 in stage 0.0 (TID 24). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 33
25/02/20 19:56:40 INFO Executor: Running task 33.0 in stage 0.0 (TID 33)
25/02/20 19:56:40 INFO Executor: Finished task 26.0 in stage 0.0 (TID 26). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 34
25/02/20 19:56:40 INFO Executor: Running task 34.0 in stage 0.0 (TID 34)
25/02/20 19:56:40 INFO Executor: Finished task 28.0 in stage 0.0 (TID 28). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 36
25/02/20 19:56:40 INFO Executor: Running task 36.0 in stage 0.0 (TID 36)
25/02/20 19:56:40 INFO Executor: Finished task 29.0 in stage 0.0 (TID 29). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 37
25/02/20 19:56:40 INFO Executor: Running task 37.0 in stage 0.0 (TID 37)
25/02/20 19:56:41 INFO Executor: Finished task 33.0 in stage 0.0 (TID 33). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 34.0 in stage 0.0 (TID 34). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 41
25/02/20 19:56:41 INFO Executor: Running task 41.0 in stage 0.0 (TID 41)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 42
25/02/20 19:56:41 INFO Executor: Running task 42.0 in stage 0.0 (TID 42)
25/02/20 19:56:41 INFO Executor: Finished task 37.0 in stage 0.0 (TID 37). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 45
25/02/20 19:56:41 INFO Executor: Running task 45.0 in stage 0.0 (TID 45)
25/02/20 19:56:41 INFO Executor: Finished task 36.0 in stage 0.0 (TID 36). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 46
25/02/20 19:56:41 INFO Executor: Running task 46.0 in stage 0.0 (TID 46)
25/02/20 19:56:41 INFO Executor: Finished task 41.0 in stage 0.0 (TID 41). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 50
25/02/20 19:56:41 INFO Executor: Running task 50.0 in stage 0.0 (TID 50)
25/02/20 19:56:41 INFO Executor: Finished task 45.0 in stage 0.0 (TID 45). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 42.0 in stage 0.0 (TID 42). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 51
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 52
25/02/20 19:56:41 INFO Executor: Running task 51.0 in stage 0.0 (TID 51)
25/02/20 19:56:41 INFO Executor: Running task 52.0 in stage 0.0 (TID 52)
25/02/20 19:56:41 INFO Executor: Finished task 46.0 in stage 0.0 (TID 46). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 55
25/02/20 19:56:41 INFO Executor: Running task 55.0 in stage 0.0 (TID 55)
25/02/20 19:56:41 INFO Executor: Finished task 50.0 in stage 0.0 (TID 50). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 57
25/02/20 19:56:41 INFO Executor: Running task 57.0 in stage 0.0 (TID 57)
25/02/20 19:56:41 INFO Executor: Finished task 52.0 in stage 0.0 (TID 52). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 51.0 in stage 0.0 (TID 51). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 61
25/02/20 19:56:41 INFO Executor: Running task 61.0 in stage 0.0 (TID 61)
25/02/20 19:56:41 INFO Executor: Finished task 55.0 in stage 0.0 (TID 55). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 57.0 in stage 0.0 (TID 57). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 62
25/02/20 19:56:41 INFO Executor: Running task 62.0 in stage 0.0 (TID 62)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 63
25/02/20 19:56:41 INFO Executor: Running task 63.0 in stage 0.0 (TID 63)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 64
25/02/20 19:56:41 INFO Executor: Running task 64.0 in stage 0.0 (TID 64)
25/02/20 19:56:41 INFO Executor: Finished task 61.0 in stage 0.0 (TID 61). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 66
25/02/20 19:56:41 INFO Executor: Running task 66.0 in stage 0.0 (TID 66)
25/02/20 19:56:41 INFO Executor: Finished task 62.0 in stage 0.0 (TID 62). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 63.0 in stage 0.0 (TID 63). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 68
25/02/20 19:56:41 INFO Executor: Running task 68.0 in stage 0.0 (TID 68)
25/02/20 19:56:41 INFO Executor: Finished task 64.0 in stage 0.0 (TID 64). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 71
25/02/20 19:56:41 INFO Executor: Running task 71.0 in stage 0.0 (TID 71)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 74
25/02/20 19:56:41 INFO Executor: Running task 74.0 in stage 0.0 (TID 74)
25/02/20 19:56:41 INFO Executor: Finished task 68.0 in stage 0.0 (TID 68). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 66.0 in stage 0.0 (TID 66). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 76
25/02/20 19:56:41 INFO Executor: Running task 76.0 in stage 0.0 (TID 76)
25/02/20 19:56:41 INFO Executor: Finished task 74.0 in stage 0.0 (TID 74). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 77
25/02/20 19:56:41 INFO Executor: Running task 77.0 in stage 0.0 (TID 77)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 78
25/02/20 19:56:41 INFO Executor: Finished task 71.0 in stage 0.0 (TID 71). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Running task 78.0 in stage 0.0 (TID 78)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 79
25/02/20 19:56:41 INFO Executor: Running task 79.0 in stage 0.0 (TID 79)
25/02/20 19:56:41 INFO Executor: Finished task 76.0 in stage 0.0 (TID 76). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 78.0 in stage 0.0 (TID 78). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 79.0 in stage 0.0 (TID 79). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 77.0 in stage 0.0 (TID 77). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 84
25/02/20 19:56:41 INFO Executor: Running task 84.0 in stage 0.0 (TID 84)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 85
25/02/20 19:56:41 INFO Executor: Running task 85.0 in stage 0.0 (TID 85)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 86
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 87
25/02/20 19:56:41 INFO Executor: Running task 87.0 in stage 0.0 (TID 87)
25/02/20 19:56:41 INFO Executor: Running task 86.0 in stage 0.0 (TID 86)
25/02/20 19:56:41 INFO Executor: Finished task 85.0 in stage 0.0 (TID 85). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 86.0 in stage 0.0 (TID 86). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 84.0 in stage 0.0 (TID 84). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 93
25/02/20 19:56:41 INFO Executor: Finished task 87.0 in stage 0.0 (TID 87). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Running task 93.0 in stage 0.0 (TID 93)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 94
25/02/20 19:56:41 INFO Executor: Running task 94.0 in stage 0.0 (TID 94)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 95
25/02/20 19:56:41 INFO Executor: Running task 95.0 in stage 0.0 (TID 95)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 97
25/02/20 19:56:41 INFO Executor: Running task 97.0 in stage 0.0 (TID 97)
25/02/20 19:56:41 INFO Executor: Finished task 94.0 in stage 0.0 (TID 94). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 93.0 in stage 0.0 (TID 93). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 95.0 in stage 0.0 (TID 95). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 97.0 in stage 0.0 (TID 97). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
25/02/20 19:56:41 INFO MemoryStore: MemoryStore cleared
25/02/20 19:56:41 INFO BlockManager: BlockManager stopped
25/02/20 19:56:41 INFO ShutdownHookManager: Shutdown hook called
25/02/20 19:56:41 INFO ShutdownHookManager: Deleting directory /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7
25/02/20 19:56:41 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7
Любая помощь будет очень оценена. < /p>
Подробнее здесь: https://stackoverflow.com/questions/794 ... ter-docker
