Работник Python неожиданно вышел из строя ⇐ Python

[Anonymous]

Код: Выделить всё

print(sc.parallelize([1, 2, 3, 4]).map(lambda x:x*x).collect())

Код: Выделить всё

Error: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 16 in stage 1.0 failed 1 times, most recent failure: Lost task 16.0 in stage 1.0 (TID 34) (RFL813 executor driver): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)   at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:612)   at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$1.applyOrElse(PythonRunner.scala:594)   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)    at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:789)    at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:766)    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:525)  at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)   at scala.collection.Iterator.foreach(Iterator.scala:943)    at scala.collection.Iterator.foreach$(Iterator.scala:943)   at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)   at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)   at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49) at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)   at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)  at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)    at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)    at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)  at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)  at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339) at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
...
at java.io.DataInputStream.readInt(Unknown Source)  at org.apache.spark.api.python.PythonRunner$$anon$3.read(PythonRunner.scala:774)    ... 32 more
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

когда я использую распараллеливание и сбор, он работает, но когда я использую распараллеливание, а затем сопоставляю, а затем собираю, появляется вышеуказанная ошибка

Подробнее здесь: https://stackoverflow.com/questions/791 ... ly-crashed