Пакет Litellm действительно поддерживает звонок сельдерея с - -pool = Gevent ⇐ Python

[Anonymous]

Я хочу выполнить параллельный призыв к модели LLM, используя пакет Litellm == 1.49.7 с использованием сельдерея с - -pool = gevent
Но я получил следующую ошибку, как мне решить < /p>
response = loop.create_task(completion(
model=SELECTED_AI_MODEL,
messages=messages,
temperature=0,
max_tokens=max_tokens,
top_p=0.1,
stream=True,
max_retries=2,
timeout=60,
response_format={"type": "json_object"},
))
< /code>
Exception in thread
[2024-10-18 20:54:12,023: WARNING/MainProcess] Thread-7 (run_success_logging_and_cache_storage)
[2024-10-18 20:54:12,023: WARNING/MainProcess] :
[2024-10-18 20:54:12,023: WARNING/MainProcess] Traceback (most recent call last):
[2024-10-18 20:54:12,023: WARNING/MainProcess] File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
[2024-10-18 20:54:12,025: WARNING/MainProcess]
[2024-10-18 20:54:12,026: WARNING/MainProcess] self.run()
[2024-10-18 20:54:12,026: WARNING/MainProcess] File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1010, in run
[2024-10-18 20:54:12,026: WARNING/MainProcess]
[2024-10-18 20:54:12,027: WARNING/MainProcess] self._target(*self._args, **self._kwargs)
[2024-10-18 20:54:12,027: WARNING/MainProcess] File "/Users/users/OFFICE_PROJECTS/ai/services/venv/lib/python3.12/site-packages/litellm/utils.py", line 7854, in run_success_logging_and_cache_storage
[2024-10-18 20:54:12,031: WARNING/MainProcess]
[2024-10-18 20:54:12,031: WARNING/MainProcess] asyncio.run(
[2024-10-18 20:54:12,031: WARNING/MainProcess] File "/usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 190, in run
[2024-10-18 20:54:12,032: WARNING/MainProcess]
[2024-10-18 20:54:12,032: WARNING/MainProcess] raise RuntimeError(
[2024-10-18 20:54:12,033: WARNING/MainProcess] RuntimeError
[2024-10-18 20:54:12,033: WARNING/MainProcess] :
[2024-10-18 20:54:12,033: WARNING/MainProcess] asyncio.run() cannot be called from a running event loop
[2024-10-18 20:54:12,034: WARNING/MainProcess] /usr/local/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py RuntimeWarning: coroutine 'Logging.async_success_handler' was never awaited
self._invoke_excepthook(self)
< /code>
any one has any ideas.
I have tried using pool=threads and pool=prefork. It worked but due to high i/o operations. I think gevent is more better than prefork and threads

Подробнее здесь: https://stackoverflow.com/questions/791 ... ool-gevent