Delta Lake для настройки ноутбуков AWS Glue ⇐ Python

[Anonymous]

Я бы хотел настроить формат Delta Lake на AWS Glue и выполнить простую ETL с помощью df.write.format ("delta"). Mode ("Overwrite"). Сохранить. (s3) может ли кто-нибудь предоставить мне код копии-паста для этого?{
"--datalake-formats": "delta"
}
< /code>
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
< /code>
and then according to documentation -

Create a key named --conf for your AWS Glue job, and set it to the following value. Alternatively, you can set the following configuration using SparkConf in your script. These settings help Apache Spark correctly handle Delta Lake tables.

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore
< /code>
import pyspark
from delta.tables import *
from delta import *
from pyspark.sql.types import *
from pyspark.sql.functions import *

# Create a spark session with Delta
builder = pyspark.sql.SparkSession.builder.appName("DeltaTutorial") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

# Create spark context
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
< /code>
And then I got the error:

ModuleNotFoundError: No module named 'delta'

or without import part
# Create a spark session with Delta
builder = pyspark.sql.SparkSession.builder.appName("DeltaTutorial") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

# Create spark context
spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
< /code>
error

NameError: name 'configure_spark_with_delta_pip' is not defined


Подробнее здесь: https://stackoverflow.com/questions/776 ... book-setup