Определение правильных версий JAR для интеграции S3 с PySpark 3.5 ⇐ Python

[Гость]

Я пытаюсь настроить среду PySpark для чтения данных из S3 с помощью PySpark 3.5 в среде Conda (Python 3.12). Однако я столкнулся с трудностями при определении правильных версий JAR-файлов aws-java-sdk и hadoop-aws (необходимых для непосредственного чтения данных из S3). Вот шаги, которые я выполнил, и проблемы, с которыми я столкнулся:

Код: Выделить всё

conda create --name spark_env
conda activate spark_env
pip install pyspark

Затем в своем блокноте Jupyter (который использует ранее созданную среду) я создаю SparkSession:

Код: Выделить всё

from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("Read from S3") \
.config("spark.hadoop.fs.s3a.access.key", "AWS_ACCESS_KEY") \
.config("spark.hadoop.fs.s3a.secret.key", "AWS_SECRET_KEY") \
.config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.getOrCreate()

s3_path = "s3a://bucket/file.json"
df = spark.read.json(s3_path)
df.show()

What gaves me the following exception:
"java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found"
That's why I have tried to download different versions of

• hadoop-aws
• aws-java-sdk

But after adding these jars to the /jars path that Pyspark is using (I have also tried specifying manually the path to these jars), I keep getting exceptions like:
"Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.impl.prefetch.PrefetchingStatistics"
or
"""
Py4JJavaError: An error occurred while calling o34.json.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 10001) (YYY executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(Lorg/apache/hadoop/fs/statistics/DurationTracker;Lorg/apache/hadoop/util/functional/CallableRaisingIOE;)Ljava/lang/Object;
or
"java.lang.ClassNotFoundException: com.amazonaws.services.s3.model.MultiObjectDeleteException"
Can someone guide me on how to identify the correct versions of the aws-java-sdk and hadoop-aws JARs to use with PySpark 3.5 for S3 integration, or provide a reliable source where this information is available? Additionally, if there's a different or more recommended approach to reading data from S3 using PySpark 3.5, I'm open to suggestions.
Thanks in advance.


Источник: https://stackoverflow.com/questions/772 ... yspark-3-5