Странная ошибка соединения Celery + Redis с задачей, которая занимает 5 минут? ⇐ Python

[Anonymous]

Я пытался отладить эту странную ошибку, из-за которой задача работника celery завершается сбоем, если задача берет на себя управление примерно через 5 минут. Он также не показывает полезную ошибку, а только следующую:

Код: Выделить всё

2024-11-14T07:25:40Z app[48e501ec74e118] den [info]Traceback (most recent call last):
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/app/trace.py", line 520, in trace_task
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    task.backend.mark_as_done(
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 157, in mark_as_done
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    self.store_result(task_id, result, state, request=request)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 526, in store_result
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    self._store_result(task_id, result, state, traceback,
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 981, in _store_result
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    self._set_with_state(self.get_key_for_task(task_id), self.encode(meta), state)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 854, in _set_with_state
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    return self.set(key, value)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/redis.py", line 400, in set
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    return self.ensure(self._set, (key, value), **retry_policy)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/redis.py", line 384, in ensure
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    return retry_over_time(
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    return fun(*args, **kwargs)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/redis.py", line 409, in _set
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    pipe.execute()
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1530, in execute
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    return conn.retry.call_with_retry(
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    return do()
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]           ^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1531, in 
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    lambda:  execute(conn, stack, raise_on_error),
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1416, in _execute_transaction
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    self.raise_first_error(commands, response)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1451, in raise_first_error
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    raise r
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1385, in _execute_transaction
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    self.parse_response(connection, "_")
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1462, in parse_response
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    result = Redis.parse_response(self, connection, command_name, **options)
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 584, in parse_response
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    response = connection.read_response()
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]               ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/connection.py", line 616, in read_response
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]    raise response
2024-11-14T07:25:40Z app[48e501ec74e118] den [info]  warn(RuntimeWarning(
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]Traceback (most recent call last):
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/billiard/pool.py", line 1794, in safe_apply_callback
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    fun(*args, **kwargs)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/worker/request.py", line 624, in on_failure
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    self.task.backend.mark_as_failure(
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 167, in mark_as_failure
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    self.store_result(task_id, exc, state,
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 526, in store_result
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    self._store_result(task_id, result, state, traceback,
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 981, in _store_result
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    self._set_with_state(self.get_key_for_task(task_id), self.encode(meta), state)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/base.py", line 854, in _set_with_state
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    return self.set(key, value)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/redis.py", line 400, in set
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    return self.ensure(self._set, (key, value), **retry_policy)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/redis.py", line 384, in ensure
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    return retry_over_time(
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    return fun(*args,  **kwargs)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/celery/backends/redis.py", line 409, in _set
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    pipe.execute()
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1530, in execute
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    return conn.retry.call_with_retry(
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    return do()
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]           ^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1531, in 
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    lambda: execute(conn, stack, raise_on_error),
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1416, in _execute_transaction
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    self.raise_first_error(commands, response)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1451, in raise_first_error
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    raise r
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1385, in _execute_transaction
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    self.parse_response(connection, "_")
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 1462, in parse_response
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    result = Redis.parse_response(self, connection, command_name, **options)
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/client.py", line 584, in parse_response
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    response = connection.read_response()
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]               ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]  File "/app/.venv/lib/python3.12/site-packages/redis/connection.py", line 616, in read_response
2024-11-14T07:25:41Z app[48e501ec74e118] den [info]    raise response

Вот задача:

Код: Выделить всё

@tracer.start_as_current_span("transcribe_runpod")
def transcribe_runpod(audio_url: str, primary_language: str) -> dict[str, str]:
log.info(f"Transcribing audio from {audio_url} with primary language {primary_language}")

response = whisper_endpoint.run_sync({
"input": {
"model": "large-v2",
"audio": audio_url,
"word_timestamps": True,
"language": language,
"logprob_threshold": None,
"no_speech_threshold": None,
"condition_on_previous_text": True,
"temperature": 0.0,
"initial_prompt": transliterated_words,
}
})

if not response or "transcription" not in response or "word_timestamps" not in response:
raise RunpodResponseEmptyError()

log.info("Transcribed audio", response=response)
return response["transcription"], response["word_timestamps"]

который вызывается

Код: Выделить всё

@celery.task(bind=True)
def transcribe_runpod_task(self, audio_url: str, primary_language: str):
return transcribe_runpod(audio_url, primary_language)

Теперь более короткие аудиофайлы работают отлично, но что-то в этом большем файле вызывает что-то вроде тайм-аута.
Эта проблема проявляется в сельдерее[ redis] 5.4.0, а также 5.5.0rc1. Мне интересно, есть ли какая-то конфигурация, которую мне не хватает?

Подробнее здесь: https://stackoverflow.com/questions/791 ... -5-minutes