Сначала я запустил туннель для служб:
Код: Выделить всё
minikube service --all
Код: Выделить всё
|-----------|------------|-------------|--------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|------------|-------------|--------------|
| default | kubernetes | | No node port |
|-----------|------------|-------------|--------------|
😿 service default/kubernetes has no node port
|-----------|----------------------|--------------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|----------------------|--------------------|---------------------------|
| default | spark-master-service | submit-port/7077 | http://192.168.67.2:30100 |
| | | ui-spark-port/8080 | http://192.168.67.2:30101 |
|-----------|----------------------|--------------------|---------------------------|
|-----------|---------------------|--------------------|---------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|---------------------|--------------------|---------------------------|
| default | spark-woker-service | ui-spark-port/8081 | http://192.168.67.2:30102 |
|-----------|---------------------|--------------------|---------------------------|
🏃 Starting tunnel for service kubernetes.
🏃 Starting tunnel for service spark-master-service.
🏃 Starting tunnel for service spark-woker-service.
|-----------|----------------------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|----------------------|-------------|------------------------|
| default | kubernetes | | http://127.0.0.1:61501 |
| default | spark-master-service | | http://127.0.0.1:61503 |
| | | | http://127.0.0.1:61504 |
| default | spark-woker-service | | http://127.0.0.1:61506 |
|-----------|----------------------|-------------|------------------------|
🎉 Opening service default/kubernetes in default browser...
[default spark-master-service http://127.0.0.1:61503
http://127.0.0.1:61504]
🎉 Opening service default/spark-woker-service in default browser...
❗ Because you are using a Docker driver on windows, the terminal needs to be open to run it.
Код: Выделить всё
from pyspark.sql import SparkSession
def main():
spark = SparkSession.builder \
.appName("PrintNumbers") \
.getOrCreate()
numbers = list(range(1, 21))
rdd = spark.sparkContext.parallelize(numbers)
rdd.foreach(print)
spark.stop()
if __name__ == "__main__":
main()
Код: Выделить всё
C:\Spark\spark-3.0.0-bin-hadoop3.2>bin\spark-submit --master k8s://http://127.0.0.1:61503 ^
More? --deploy-mode cluster ^
More? --conf spark.executor.instances=1 ^
More? --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark ^
More? --conf spark.kubernetes.container.image=bitnami/spark ^
More? --class org.apache.spark.examples.JavaSparkPi ^
More? --name spark-pi ^
More? "local:///D:/spark-k8s tutorial/spark-submit.py"
Код: Выделить всё
23/11/19 19:10:35 WARN Utils: Kubernetes master URL uses HTTP instead of HTTPS.
23/11/19 19:10:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/11/19 19:10:35 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
23/11/19 19:10:36 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
23/11/19 19:10:36 WARN WatchConnectionManager: Exec Failure
java.io.IOException: unexpected end of stream on Connection{127.0.0.1:61503, proxy=DIRECT hostAddress=/127.0.0.1:61503 cipherSuite=none protocol=http/1.1}
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:111)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=à
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:236)
at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
... 28 more
Exception in thread "kubernetes-dispatcher-0" Exception in thread "main" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@69348aff rejected from java.util.concurrent.ScheduledThreadPoolExecutor@369a631[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:305)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:50)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:218)
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:209)
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.lang.Throwable: waiting here
at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:144)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:341)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:755)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:739)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:70)
at org.apache.spark.deploy.k8s.submit.Client.$anonfun$run$1(KubernetesClientApplication.scala:129)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2538)
at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:129)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:221)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:215)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2539)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:215)
at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:188)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: unexpected end of stream on Connection{127.0.0.1:61503, proxy=DIRECT hostAddress=/127.0.0.1:61503 cipherSuite=none protocol=http/1.1}
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:111)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
... 4 more
Caused by: java.io.EOFException: \n not found: limit=0 content=?
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:236)
at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
... 28 more
23/11/19 19:10:36 INFO ShutdownHookManager: Shutdown hook called
23/11/19 19:10:36 INFO ShutdownHookManager: Deleting directory C:\Users\ABC\AppData\Local\Temp\spark-3e986567-7acd-4561-bf9b-014183f85c64
Может кто-нибудь мне помочь? Большое спасибо.
Я пытался отправить приложение Spark, но что-то не так, и теперь мне нужна помощь по описанной моей проблеме.
Подробнее здесь: https://stackoverflow.com/questions/775 ... -on-window
