Как сделать журналы пакетных заданий доступными, когда задания выполняются внутри эфемерных контейнеров Docker? ⇐ Python

[Anonymous]

Context
So, basically I am running a cron job (python ETL script) via a docker container. That means, every day at 12.30 am my cron job runs

docker run $IMAGE In the Dockerfile I have the script like

# Run the script at container boot time. CMD ["./run_manager.sh"] This is how the ```run_manager.sh`` looks like.

python3 main.py>>main.log 2>&1 I am using the python logging module like this

#!/usr/bin/env python3 # encoding: utf-8 """ This file contains the script """ import logging from contextlib import AbstractContextManager import polars as pl import tensorflow as tf import sqlalchemy as sa logging.basicConfig(format='%(asctime)s|%(levelname)s: %(message)s', datefmt='%H:%M:%S, %d-%b-%Y', level=logging.INFO) ... # Other codes Question Since the container is an ephemeral one that is created and destroyed every day as the cron is triggered, I have no way to access the log. So how do we change it to make the logs persist, rotate and visible outside the container? Is there a way?
Addendum
Right now it is running as a cron on an on-prem Ubuntu instance. But I am going to migrate it to google cloud scheduler very soon, keeping the design intact as much as possible. Is there any solution in that case as well, basically, to be able to see the logs of past jobs?


Источник: https://stackoverflow.com/questions/780 ... ral-docker