Мобильная версияI am trying to:
- Load an SSL certification from S3 to a cluster.
- addFile so all nodes see the file.
- Create a connection URL to IBM db2 with JDBC.
Step 1 and Step 2 are working successfully. I can open the file with with open(cert_filepath, "r") as file and print it.
But in Step 3 I get the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (10.239.124.48 executor 1): com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2043][11550][4.27.25] Exception java.io.FileNotFoundException: Error opening socket to server / on port 53,101 with message: /local_disk0/spark-c30d1a7f-deea-4184-a9f9-2b6c9eab6c5e/userFiles-761620cf-2cb1-4623-9677-68694f0e4b3c/dwt01_db2_ssl.arm (No such file or directory). ERRORCODE=-4499, SQLSTATE=08001
The port is 53101 but the error code puts a comma.
The essential part of code:
sc = SparkContext.getOrCreate() s3_client = boto3.client("s3") s3_client.download_file( Bucket="my-bucket-s3", Key="my/path/db2/dwt01_db2_ssl.arm", Filename="dwt01_db2_ssl.arm", ) sc.addFile("dwt01_db2_ssl.arm") cert_filepath = SparkFiles.get("dwt01_db2_ssl.arm") user_name = cls.get_aws_secret(secret_name=cls._db2_username_aws_secret_name, key="username", region="eu-central-1") password = cls.get_aws_secret(secret_name=cls._db2_password_aws_secret_name, key="password", region="eu-central-1") driver = "com.ibm.db2.jcc.DB2Driver" jdbc_url = f"jdbc:db2://{hostname}:{port}/{database}:sslConnection=true;sslCertLocation={cert_filepath};" df = ( spark.read.format("jdbc") .option("driver", driver) .option("url", jdbc_url) .option("dbtable", f"({query}) as src") .option("user", user_name) .option("password", password) .load() ) I can't seem to solve what could be the cause for this FileNotFoundException as it is there for at least when reading it and printing it.
Источник: https://stackoverflow.com/questions/780 ... lenotfound
