Код: Выделить всё
spark-submit --master yarn --deploy-mode client --executor-memory 5g --executor-cores 4 /home/hadoop/spark_data/save_data_db.py
< /code>
мой файл задания save_data_db.py:
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("InsertIntoHiveYarn") \
.enableHiveSupport() \
.config("hive.server2.authentication", "CUSTOM") \
.config("javax.jdo.option.ConnectionUserName", "hive_user") \
.config("javax.jdo.option.ConnectionPassword", "p") \
.getOrCreate()
data = [(1, "Alice", "HR"),
(2, "Bob", "Engineering"),
(3, "Charlie", "Marketing")]
df = spark.createDataFrame(data, ["id", "name", "department"])
spark.sql('USE db_test')
df.write.insertInto("employees", overwrite=False)
spark.stop()
< /code>
Я поместил файл JARS в HDF, чтобы избежать любой ошибки JAR в отдельном каталоге: < /p>
[hadoop@RA-DN1 spark]$ hdfs dfs -ls /spark-jars/
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2025-06-12 05:44 /spark-jars/hive
drwxr-xr-x - hadoop supergroup 0 2025-06-12 05:19 /spark-jars/spark
[hadoop@RA-DN1 spark]$ hdfs dfs -ls /spark-jars/hive/
Found 1 items
-rw-r--r-- 3 hadoop supergroup 258269977 2025-06-12 05:44 /spark-jars/hive/hive-libs.zip
[hadoop@RA-DN1 spark]$ hdfs dfs -ls /spark-jars/spark/
Found 1 items
-rw-r--r-- 3 hadoop supergroup 354049683 2025-06-12 05:19 /spark-jars/spark/spark-libs.zip
< /code>
spark-default.confspark.master yarn
spark.submit.deployMode cluster
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///spark-logs
spark.history.fs.logDirectory hdfs:///spark-logs
spark.yarn.historyServer.address namenode2:18080
spark.executor.memory 10g
spark.driver.memory 2g
spark.executor.instances 4
spark.executor.cores 4
spark.hadoop.fs.defaultFS hdfs://mycluster
spark.hadoop.dfs.nameservices mycluster
spark.hadoop.dfs.ha.namenodes.mycluster namenode1,namenode2
spark.hadoop.dfs.namenode.rpc-address.mycluster.namenode1 namenode1:8020
spark.hadoop.dfs.namenode.rpc-address.mycluster.namenode2 namenode2:8020
spark.hadoop.dfs.client.failover.proxy.provider.mycluster org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
spark.yarn.archive hdfs:///spark-jars/spark/spark-libs.zip
spark.kerberos.access.hadoopFileSystems hdfs://mycluster
spark.hadoop.yarn.resourcemanager.ha.enabled true
spark.hadoop.yarn.resourcemanager.cluster-id yarn-cluster
spark.hadoop.yarn.resourcemanager.ha.rm-ids rm1,rm2
spark.hadoop.yarn.resourcemanager.hostname.rm1 namenode1
spark.hadoop.yarn.resourcemanager.hostname.rm2 namenode2
spark.hadoop.yarn.resourcemanager.address.rm1 namenode1:8032
spark.hadoop.yarn.resourcemanager.address.rm2 namenode2:8032
spark.hadoop.yarn.resourcemanager.scheduler.address.rm1 namenode1:8030
spark.hadoop.yarn.resourcemanager.scheduler.address.rm2 namenode2:8030
spark.sql.catalogImplementation=hive
spark.sql.warehouse.dir=hdfs://mycluster/user/hive/warehouse2
spark.hadoop.hive.metastore.uris=thrift://10.101.10.20:9083
spark.sql.hive.metastore.version=3.1.3
spark.sql.hive.metastore.jars=hdfs:///spark-jars/hive/hive-libs.zip
< /code>
Spark submission error:
with ID 4, ResourceProfileId 0
2025-06-12 06:05:56 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
2025-06-12 06:05:56 INFO BlockManagerMasterEndpoint: Registering block manager datanode2:43323 with 2.5 GiB RAM, BlockManagerId(4, datanode2, 43323, None)
2025-06-12 06:05:56 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
2025-06-12 06:05:56 INFO SharedState: Warehouse path is 'hdfs://mycluster/user/hive/warehouse2'.
2025-06-12 06:05:56 INFO ServerInfo: Adding filter to /SQL: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2025-06-12 06:05:56 INFO ServerInfo: Adding filter to /SQL/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2025-06-12 06:05:56 INFO ServerInfo: Adding filter to /SQL/execution: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2025-06-12 06:05:56 INFO ServerInfo: Adding filter to /SQL/execution/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2025-06-12 06:05:56 INFO ServerInfo: Adding filter to /static/sql: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2025-06-12 06:05:59 INFO HiveUtils: Initializing HiveMetastoreConnection version 3.1.3 using file:/home/hadoop/spark_data/hdfs:file:/spark-jars/hive/hive-libs.zip
Traceback (most recent call last):
File "/home/hadoop/spark_data/save_data_db.py", line 17, in
spark.sql('USE db_test')
File "/data/hadoop/binary/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1631, in sql
File "/data/hadoop/binary/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/data/hadoop/binary/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
File "/data/hadoop/binary/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o62.sql.
: java.lang.NoClassDefFoundError: o r g / a p a c h e / h a d o o p / h i v e / m e t a s t o r e / a p i / A l r e a d y E x i s t s E x c e p t i o n < b r / > a t j a v a . b a s e / j a v a . l a n g . C l a s s . g e t D e c l a r e d C o n s t r u c t o r s 0 ( N a t i v e M e t h o d ) < b r / > a t j a v a . b a s e / j a v a . l a n g . C l a s s . p r i v a t e G e t D e c l a r e d C o n s t r u c t o r s ( C l a s s . j a v a : 3 1 3 7 ) < b r / > a t j a v a . b a s e / j a v a . l a n g . C l a s s . g e t C o n s t r u c t o r s ( C l a s s . j a v a : 1 9 4 3 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . c l i e n t . I s o l a t e d C l i e n t L o a d e r . c r e a t e C l i e n t ( I s o l a t e d C l i e n t L o a d e r . s c a l a : 3 1 7 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e U t i l s $ . n e w C l i e n t F o r M e t a d a t a ( H i v e U t i l s . s c a l a : 4 8 8 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e U t i l s $ . n e w C l i e n t F o r M e t a d a t a ( H i v e U t i l s . s c a l a : 3 7 6 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e E x t e r n a l C a t a l o g . c l i e n t $ l z y c o m p u t e ( H i v e E x t e r n a l C a t a l o g . s c a l a : 6 8 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e E x t e r n a l C a t a l o g . c l i e n t ( H i v e E x t e r n a l C a t a l o g . s c a l a : 6 7 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e E x t e r n a l C a t a l o g . $ a n o n f u n $ d a t a b a s e E x i s t s $ 1 ( H i v e E x t e r n a l C a t a l o g . s c a l a : 2 2 4 ) < b r / > a t s c a l a . r u n t i m e . j a v a 8 . J F u n c t i o n 0 $ m c Z $ s p . a p p l y ( J F u n c t i o n 0 $ m c Z $ s p . j a v a : 2 3 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e E x t e r n a l C a t a l o g . w i t h C l i e n t ( H i v e E x t e r n a l C a t a l o g . s c a l a : 9 9 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e E x t e r n a l C a t a l o g . d a t a b a s e E x i s t s ( H i v e E x t e r n a l C a t a l o g . s c a l a : 2 2 4 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . S h a r e d S t a t e . e x t e r n a l C a t a l o g $ l z y c o m p u t e ( S h a r e d S t a t e . s c a l a : 1 4 6 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . S h a r e d S t a t e . e x t e r n a l C a t a l o g ( S h a r e d S t a t e . s c a l a : 1 4 0 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . S h a r e d S t a t e . g l o b a l T e m p V i e w M a n a g e r $ l z y c o m p u t e ( S h a r e d S t a t e . s c a l a : 1 7 6 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . i n t e r n a l . S h a r e d S t a t e . g l o b a l T e m p V i e w M a n a g e r ( S h a r e d S t a t e . s c a l a : 1 7 4 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e S e s s i o n S t a t e B u i l d e r . $ a n o n f u n $ c a t a l o g $ 2 ( H i v e S e s s i o n S t a t e B u i l d e r . s c a l a : 7 0 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . c a t a l y s t . c a t a l o g . S e s s i o n C a t a l o g . g l o b a l T e m p V i e w M a n a g e r $ l z y c o m p u t e ( S e s s i o n C a t a l o g . s c a l a : 1 2 4 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . c a t a l y s t . c a t a l o g . S e s s i o n C a t a l o g . g l o b a l T e m p V i e w M a n a g e r ( S e s s i o n C a t a l o g . s c a l a : 1 2 4 ) < b r / > a t o r g . a p a c h e . s p a r k . s q l . c a t a l y s t . c a t a l o g . S e s s i o n C a t a l o g . s e t C u r r e n t D a t a b a seWithNameCheck(SessionCatalog.scala:340)
at org.apache.spark.sql.connector.catalog.CatalogManager.setCurrentNamespace(CatalogManager.scala:119)
at org.apache.spark.sql.execution.datasources.v2.SetCatalogAndNamespaceExec.$anonfun$run$2(SetCatalogAndNamespaceExec.scala:36)
at org.apache.spark.sql.execution.datasources.v2.SetCatalogAndNamespaceExec.$anonfun$run$2$adapted(SetCatalogAndNamespaceExec.scala:36)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.sql.execution.datasources.v2.SetCatalogAndNamespaceExec.run(SetCatalogAndNamespaceExec.scala:36)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:107)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:85)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:83)
at org.apache.spark.sql.Dataset.(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:638)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:629)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:659)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.AlreadyExistsException
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:274)
at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:263)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
... 68 more
2025-06-12 06:05:59 INFO SparkContext: Invoking stop() from shutdown hook
2025-06-12 06:05:59 INFO SparkContext: SparkContext is stopping with exitCode 0.
2025-06-12 06:05:59 INFO SparkUI: Stopped Spark web UI at http://datanode1:4040
2025-06-12 06:05:59 INFO YarnClientSchedulerBackend: Interrupting monitor thread
2025-06-12 06:05:59 INFO YarnClientSchedulerBackend: Shutting down all executors
2025-06-12 06:05:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
2025-06-12 06:05:59 INFO YarnClientSchedulerBackend: YARN client scheduler backend Stopped
2025-06-12 06:05:59 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
2025-06-12 06:05:59 INFO MemoryStore: MemoryStore cleared
2025-06-12 06:05:59 INFO BlockManager: BlockManager stopped
2025-06-12 06:05:59 INFO BlockManagerMaster: BlockManagerMaster stopped
2025-06-12 06:05:59 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
2025-06-12 06:05:59 INFO SparkContext: Successfully stopped SparkContext
2025-06-12 06:05:59 INFO ShutdownHookManager: Shutdown hook called
2025-06-12 06:05:59 INFO ShutdownHookManager: Deleting directory /tmp/spark-4c9908be-eea2-4e85-95a4-d20a87f0cb94
2025-06-12 06:05:59 INFO ShutdownHookManager: Deleting directory /tmp/spark-4c9908be-eea2-4e85-95a4-d20a87f0cb94/pyspark-1fafb003-619d-4387-8b16-433e237ecceb
2025-06-12 06:05:59 INFO ShutdownHookManager: Deleting directory /tmp/spark-8ac364fb-502d-42bd-8808-b261a7d9a693
< /code>
I am using Hive to store data from Spark into Hive. I have tried both cluster and client mode get same result.
Подробнее здесь: https://stackoverflow.com/questions/796 ... found-what
