Ошибка 404 при вызове операции HeadObject в коде Python AWS Lambda при взаимодействии с корзинами S3 ⇐ Python

[Anonymous]

Мне нужны рекомендации относительно моей лямбды, которая считывает файл из корзины S3, преобразует файл в PDF и загружает его в другую корзину S3.
Лямбда — это созданный с использованием стека Cloudformation, это файл YAML:
AWSTemplateFormatVersion: 2010-09-09
Description: Lambda template
Parameters:
Environment:
Description: Environment
Type: String
Default: test
AllowedValues: [prod, test, staging]
BucketPrefix:
Type: String
Default: crmpicco
Description: The prefix to use for the input and output buckets.
LambdaFunctionUri:
Type: String
Description: The ECR URI for the Lambda function that converts files to PDF.

Resources:
InputS3Bucket:
Type: AWS::S3::Bucket
DependsOn: LambdaInvokePermission
Properties:
BucketName: !Join [ '-', [!Ref BucketPrefix, 'input'] ]
AccessControl: Private
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: true
LifecycleConfiguration:
Rules:
- Status: Enabled
Transitions:
- StorageClass: INTELLIGENT_TIERING
TransitionInDays: '0'
NotificationConfiguration:
LambdaConfigurations:
- Event: 's3:ObjectCreated:*'
Function: !GetAtt LambdaFunction.Arn
OutputS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Join [ '-', [!Ref BucketPrefix, 'output'] ]
AccessControl: Private
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: true
LifecycleConfiguration:
Rules:
- Status: Enabled
Transitions:
- StorageClass: INTELLIGENT_TIERING
TransitionInDays: '0'

LambdaRole:
Type: AWS::IAM::Role
Description: Lambda Converter Role
Properties:
RoleName: LambdaConvertRole
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: ''
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole

LambdaPolicy:
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:*
Resource: arn:aws:logs:*:*:*
- Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
- s3:DeleteObject
- s3:HeadObject
Resource:
- !Join [ '', [ !GetAtt InputS3Bucket.Arn, '/*' ] ] # Updated to allow access to all objects
- !Join [ '', [ !GetAtt OutputS3Bucket.Arn, '/*' ] ] # Updated to allow access to all objects
PolicyName: LambdaConvertPolicy
Roles:
- !Ref LambdaRole

LambdaFunction:
Type: AWS::Lambda::Function
Description: Lambda Function to convert documents
Properties:
Architectures:
- arm64
Role: !GetAtt LambdaRole.Arn
Code:
ImageUri: !Ref LambdaFunctionUri
Environment:
Variables:
OutputBucket: !Join [ '-', [!Ref BucketPrefix, 'output'] ]
PackageType: Image
FunctionName: LambdaConvert
MemorySize: 256
Timeout: 600

LambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref LambdaFunction
Principal: s3.amazonaws.com
SourceArn: !Join [ '', ['arn:aws:s3:::', !Ref BucketPrefix, '-', 'input'] ]

Outputs:
InputBucket:
Description: S3 Input Bucket
Value: !Ref InputS3Bucket
OutputBucket:
Description: S3 Output Bucket
Value: !Ref OutputS3Bucket

Правильные ли разрешения применяются к рассматриваемой роли/политике/корпусу? Я постоянно получаю ошибку HTTP 404, однако вижу файл во «входной» корзине.
Вот какую ошибку я получаю:
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not FoundLAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/late ... pdate.html

[ERROR] ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
Traceback (most recent call last):
File "/var/task/lambdaconvert.py", line 193, in lambda_handler
raise last_exception
File "/var/task/lambdaconvert.py", line 184, in lambda_handler
process(record)
File "/var/task/lambdaconvert.py", line 210, in process
targetformat, conversionid, sourcefileid = get_object_data(bucket, key)
File "/var/task/lambdaconvert.py", line 42, in get_object_data
response = s3_client.head_object(Bucket=bucket, Key=key)
File "/var/lang/lib/python3.12/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/lang/lib/python3.12/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)

Это выполняемый код Python:
import boto3
import os
import logging
import uuid
import subprocess
from multiprocessing import Process, Pipe

s3_client = boto3.client('s3')
logger = logging.getLogger()

SOFFICE = '/usr/local/lib/libreoffice/program/soffice.bin' # Libre conversion executable.

keep_files = not os.environ.get('KEEP_FILES', '') == ''

def get_object_data(bucket, key):
"""
Get object data that will be used in the object processing.
"""

logger.info("Processing object with key {}".format(key))

response = s3_client.head_object(Bucket=bucket, Key=key)

metadata = response.get('Metadata')
if not metadata:
raise RuntimeError("Head response object without Metadata", response)

targetformat = metadata['targetformat']
conversionid = metadata['id']
sourcefileid = metadata['sourcefileid']

return (targetformat, conversionid, sourcefileid)

def process_convert_file(download_path, targetformat):
"""
Wrapper for convert_file to handle retries and exceptions based on status codes.
"""
if not check_tmp_write_access():
logger.error("No write access to /tmp directory")

status_code = convert_file(download_path, targetformat)
if status_code == 0:
logger.info("Conversion successful.")
elif status_code == 81:
logger.warning("Conversion failed with status 81, retrying once.")
status_code = convert_file(download_path, targetformat) # Retry once
if status_code != 0:
raise Exception(f"Retry failed with status code: {status_code}")
else:
raise Exception(f"Conversion failed with status code: {status_code}")

def convert_file(filepath, targetformat):
"""
Convert the input file to PDF.
Return the path of the converted document/
"""
logger.debug("Starting file conversion")
logger.debug(f"Input file: {filepath}, Target format: {targetformat}")
commandargs = [
SOFFICE,
"--headless",
"--invisible",
"--nodefault",
"--nofirststartwizard",
"--nolockcheck",
"--nologo",
"--norestore",
"--writer",
"--convert-to",
targetformat,
"--outdir",
"/tmp",
filepath # Needs to be the absolute path as a string
]

env = os.environ.copy()
env['HOME'] = '/tmp' # Set home to /tmp to avoid permission issues.

logger.debug(f"Command arguments: {commandargs}")
logger.debug(f"Environment variables: {env}")

try:
subprocess.run(commandargs, env=env, timeout=300, check=True)
logger.info("File conversion succeeded")
return 0 # Success
except subprocess.CalledProcessError as e:
logger.error(f"File conversion failed with error code: {e.returncode}")
logger.exception("Exception occurred during file conversion")
return e.returncode # Return the error code if subprocess fails
# TODO: add some logging.

def check_tmp_write_access():
test_filepath = "/tmp/test_write_access.txt"

try:
# Try writing a file to the /tmp directory
with open(test_filepath, 'w') as test_file:
test_file.write("Testing write access to /tmp")

# If the file was created, log success and delete it
if os.path.exists(test_filepath):
logger.info(f"Write access to /tmp is working. Test file created at: {test_filepath}")
# os.remove(test_filepath) # Clean up after test
return True
else:
logger.warning(f"Test file could not be created at {test_filepath}")
return False

except Exception as e:
logger.error(f"Failed to write to /tmp: {e}")
return False

def action_multiprocessing(multiprocesses):
"""
Process multiple actions at once.
This is just a thin wrapper around multiprocessing.Process.
Lambda has a limited environment so we can only use select multiprocessing tools.
We pass in a list of dictionaries of the methods we want to run along with their
args and kwargs and this method will "queue" them up and then execute them.
"""
processes = [] # create a list to keep all processes
parent_connections = [] # create a list to keep connections

for multiprocess in multiprocesses:
parent_conn, child_conn = Pipe() # Create a pipe for communication.
parent_connections.append(parent_conn)
process = Process(
target=multiprocess['method'],
args=multiprocess['processargs'],
kwargs=multiprocess['processkwargs'],
)
processes.append(process)

# Start all processes.
for process in processes:
process.start()

# Make sure that all processes have finished.
for process in processes:
process.join()

def lambda_handler(event, context):
"""
lambda_handler is the entry point that is invoked when the lambda function is called,
more information can be found in the docs:
https://docs.aws.amazon.com/lambda/late ... types.html

Get the input document from the input S3 bucket.
Convert the input file into the desired format.
Upload the converted document to the output S3 bucket.
"""

# Set logging, default to ERROR (40). DEBUG (10) is the lowest level.
logging_level = os.environ.get('LoggingLevel', logging.ERROR)
logger.setLevel(int(logging_level))

last_exception = None
try:
for record in event['Records']:
try:
process(record)
except Exception as e:
logger.exception("%r", record)
last_exception = e
except Exception:
logger.exception("%r %r", event, context)
raise

if last_exception:
raise last_exception

def process(record):
"""
Get and process the file from the input bucket.
"""

bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']

# Filter out permissions check file.
# This is initiated by Moodle to check bucket access is correct
if key == 'permissions_check_file':
return

# Get object data.
targetformat, conversionid, sourcefileid = get_object_data(bucket, key)

download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
# Conversion replaces the file extension with .pdf.
# Split the original filename from its last extension.
base_name, _ = os.path.splitext(download_path)
# Append .pdf to the base name.
upload_path = f"{base_name}.pdf"

# First multiprocessing split.
# Download LibreOffice and input bucket object.
multiprocesses = (
{
'method': s3_client.download_file,
'processargs': (bucket, key, download_path,),
'processkwargs': {}
},
)
action_multiprocessing(multiprocesses)

# Second multiprocessing split.
# Convert file and remove original from input bucket.
multiprocesses = (
{
'method': process_convert_file,
'processargs': (download_path, targetformat,),
'processkwargs': {}
},
)
if not keep_files:
multiprocesses += (
{
'method': s3_client.delete_object,
'processargs': (),
'processkwargs': {'Bucket': bucket, 'Key': key}
},
)
action_multiprocessing(multiprocesses)

# Third multiprocessing split.
# Upload converted file to output bucket and delete local input.
metadata = {"Metadata": {"id": conversionid, "sourcefileid": sourcefileid}}
multiprocesses = (
{
'method': s3_client.upload_file,
'processargs': (upload_path, os.environ['OutputBucket'], key,),
'processkwargs': {'ExtraArgs': metadata}
},
{
'method': os.remove,
'processargs': (download_path,),
'processkwargs': {}
},
)
action_multiprocessing(multiprocesses)

# Delete local output file.
os.remove(upload_path)


Подробнее здесь: https://stackoverflow.com/questions/790 ... ode-when-i