Обслуживание MMCV/MMDet на Databricks — GLIBC_2.32 не найден ⇐ Python

[Anonymous]

Я пытаюсь разместить модель MMDetection на сервере Databricks Serving (в Azure). Модель обучена на 15.4 LTS ML. Однако во время обновления конечной точки обслуживания он жалуется на GLIBC_2.32:
An error occurred while loading the model: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/_ext.cpython-311-x86_64-linux-gnu.so)

Обычно я просто запускаю apt-get install glibc или что-то в этом роде в сценарии запуска. Однако в документации Databricks указано, что сценарии инициализации не поддерживаются.
Вопрос
Итак, как обойти эту ошибку и развернуть MMDet модель на Databrciks?
Ранее я развернул множество моделей PyTorch Lightning (то есть не использовал OpenMMLab), и glibc не был проблемой для обслуживания Databricks.
Эскизы решений
Я вижу два возможных пути к потенциальному решению.

• Может быть, есть волшебный трюк разрешить установку отсутствующих библиотек на обслуживающей конечной точке?
• В качестве альтернативы, возможно, мы сможем избавиться от основной причины проблемы, а именно mmcv, и эта ошибка немного похожа на opencv проблема для меня? Модель, которую я пытаюсь использовать, — это CoDETR с конфигурацией, основанной на этой.

Оба варианта кажутся мне одинаково маловероятными. Может быть, есть другой способ?
Выдержка из использованных пакетов ключей:
pip install --upgrade pip
pip install uv
uv pip install torch==2.1.0 torchvision==0.16.0 numpy==1.26.4 openmim "mmengine==0.10.5"
uv pip install "mmcv==2.1.0" -f https://download.openmmlab.com/mmcv/dis ... index.html
uv pip install albumentations==1.4.18 pycocotools==2.0.7 mlflow python-snappy==0.7.3
uv pip install lightning==2.2.2 mmdet==3.3.0

Полная трассировка стека:
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure
[mlrpj] [2024-12-30 09:50:06 +0000] model = load_model_fn(path)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 309, in wrapper
[mlrpj] [2024-12-30 09:50:06 +0000] is_func_called, result = True, f(*args, **kwargs)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1067, in load_model
[mlrpj] [2024-12-30 09:50:06 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 561, in _load_pyfunc
[mlrpj] [2024-12-30 09:50:06 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 555, in _load_context_model_and_signature
[mlrpj] [2024-12-30 09:50:06 +0000] python_model.load_context(context=context)
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/computer_vision/engine/mmdet/utils.py", line 44, in load_context
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.apis import DetInferencer
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/apis/__init__.py", line 2, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .det_inferencer import DetInferencer
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/apis/det_inferencer.py", line 22, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.evaluation import INSTANCE_OFFSET
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/evaluation/__init__.py", line 4, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .metrics import * # noqa: F401,F403
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/evaluation/metrics/__init__.py", line 5, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .coco_metric import CocoMetric
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/evaluation/metrics/coco_metric.py", line 16, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.datasets.api_wrappers import COCO, COCOeval, COCOevalMP
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/__init__.py", line 31, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .utils import get_loading_pipeline
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/utils.py", line 5, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.datasets.transforms import LoadAnnotations, LoadPanopticAnnotations
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/transforms/__init__.py", line 6, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .formatting import (ImageToTensor, PackDetInputs, PackReIDInputs,
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/datasets/transforms/formatting.py", line 11, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.structures.bbox import BaseBoxes
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/bbox/__init__.py", line 2, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .base_boxes import BaseBoxes
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/bbox/base_boxes.py", line 9, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmdet.structures.mask.structures import BitmapMasks, PolygonMasks
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/mask/__init__.py", line 3, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .structures import (BaseInstanceMasks, BitmapMasks, PolygonMasks,
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmdet/structures/mask/structures.py", line 12, in
[mlrpj] [2024-12-30 09:50:06 +0000] from mmcv.ops.roi_align import roi_align
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/ops/__init__.py", line 3, in
[mlrpj] [2024-12-30 09:50:06 +0000] from .active_rotated_filter import active_rotated_filter
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in
[mlrpj] [2024-12-30 09:50:06 +0000] ext_module = ext_loader.load_ext(
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
[mlrpj] [2024-12-30 09:50:06 +0000] ext = importlib.import_module('mmcv.' + name)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/importlib/__init__.py", line 126, in import_module
[mlrpj] [2024-12-30 09:50:06 +0000] return _bootstrap._gcd_import(name[level:], package, level)
[mlrpj] [2024-12-30 09:50:06 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:06 +0000] ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/_ext.cpython-311-x86_64-linux-gnu.so)
[mlrpj] [2024-12-30 09:50:06 +0000] [595] [INFO] Worker exiting (pid: 595)
[mlrpj] [2024-12-30 09:50:07 +0000] An error occurred while loading the model: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mmcv/_ext.cpython-311-x86_64-linux-gnu.so)
[mlrpj] [2024-12-30 09:50:07 +0000] Traceback (most recent call last):
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 212, in get_model_option_or_exit
[mlrpj] [2024-12-30 09:50:07 +0000] self.model = self.model_future.result()
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result
[mlrpj] [2024-12-30 09:50:07 +0000] return self.__get_result()
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
[mlrpj] [2024-12-30 09:50:07 +0000] raise self._exception
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/concurrent/futures/thread.py", line 58, in run
[mlrpj] [2024-12-30 09:50:07 +0000] result = self.fn(*self.args, **self.kwargs)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflowserving/scoring_server/__init__.py", line 130, in _load_model_closure
[mlrpj] [2024-12-30 09:50:07 +0000] model = load_model_fn(path)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/tracing/provider.py", line 309, in wrapper
[mlrpj] [2024-12-30 09:50:07 +0000] is_func_called, result = True, f(*args, **kwargs)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/__init__.py", line 1067, in load_model
[mlrpj] [2024-12-30 09:50:07 +0000] model_impl = importlib.import_module(conf[MAIN])._load_pyfunc(data_path)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 561, in _load_pyfunc
[mlrpj] [2024-12-30 09:50:07 +0000] context, python_model, signature = _load_context_model_and_signature(model_path, model_config)
[mlrpj] [2024-12-30 09:50:07 +0000] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/mlflow/pyfunc/model.py", line 555, in _load_context_model_and_signature
[mlrpj] [2024-12-30 09:50:07 +0000] python_model.load_context(context=context)
[mlrpj] [2024-12-30 09:50:07 +0000] File "/opt/conda/envs/mlflow-env/lib/python3.11/site-packages/computer_vision/engine/mmdet/utils.p


Подробнее здесь: https://stackoverflow.com/questions/793 ... -not-found