Как точно расположить 3D-ограничивающие рамки в Unity для HoloLens 2 на основе результатов обнаружения 2D-объектов? ⇐ C#

[Anonymous]

I am developing an application for HoloLens 2 using Unity 2022.3.30f1 and MRTK Foundation 2.8.3. My goal is to position 3D bounding boxes in the real world based on 2D object detection results obtained from a YOLOv9 server.
Problem:

Несмотря на несколько попыток, ограничивающие рамки либо расположены неточно, либо перекрывают друг друга, если обнаружено несколько объектов. Размеры ограничивающих рамок также неверны.
Код для отображения 2D-координат с сервера yolov9 в 3D-мировое пространство в единстве:
private void CreateBoundingBox(обнаружение обнаружения)
{
Matrix4x4 проекцияMatrix = Camera.main.projectionMatrix;
Matrix4x4 cameraMatrix = Camera.main.worldToCameraMatrix;
Matrix4x4 vpMatrix = projectionMatrix * cameraMatrix;
Matrix4x4 vpMatrixInverse = vpMatrix.inverse;
Vector3 bottomLeftViewport = new Vector3(detection.x1 / inputImage.width, detection.y1 / inputImage.height, 0.5f);
Vector3 topRightViewport = new Vector3(detection.x2 / inputImage.width, detection.y2 / inputImage.height, 0.5f);

Vector3 bottomLeftWorld = GetWorldCoordinatesFromViewport(bottomLeftViewport, vpMatrixInverse);
Vector3 topRightWorld = GetWorldCoordinatesFromViewport(topRightViewport, vpMatrixInverse);

Vector3 centerWorld = (bottomLeftWorld + topRightWorld) / 2;
Vector3 sizeWorld = topRightWorld - bottomLeftWorld;

detectionResultsText.text += $"bottom-left viewport: {bottomLeftViewport}\n";
detectionResultsText.text += $"top-right viewport: {topRightViewport}\n";
detectionResultsText.text += $"bottom-left world: {bottomLeftWorld}\n";
detectionResultsText.text += $"top-right world: {topRightWorld}\n";
detectionResultsText.text += $"center world: {centerWorld}\n";
detectionResultsText.text += $"size world: {sizeWorld}\n";

BoundingBoxHolder boundingBoxHolder = Instantiate(boundingBoxHolderPrefab, centerWorld, Quaternion.identity);
boundingBoxHolder.SetBoundingBoxSize(sizeWorld);
boundingBoxHolder.SetClassText(detection.@class);
boundingBoxHolder.transform.LookAt(Camera.main.transform);

boundingBoxHolders.Add(boundingBoxHolder);
}

Information
Image Dimensions:

Captured Image Dimensions: 2272 x 1278
Processed Image Dimensions by YOLOv9 server: 2272 x 1278
Camera Specifications:

FOV: 60 degrees
Near Plane: 0.1
Far Plane: 1000
Example JSON from YOLOv9 Server: Multiple objects that are detected
`{
"objects": [
{"class": "chair", "confidence": 0.8503433465957642, "x1": 40.0, "y1": 12.0, "x2": 184.0, "y2": 132.0},
{"class": "bottle", "confidence": 0.811337411403656, "x1": 80.0, "y1": 74.0, "x2": 117.0, "y2": 150.0},
{"class": "backpack", "confidence": 0.7879111766815186, "x1": 269.0, "y1": 137.0, "x2": 353.0, "y2": 245.0},
{"class": "bottle", "confidence": 0.4553461968898773, "x1": 70.0, "y1": 98.0, "x2": 102.0, "y2": 162.0},
{"class": "keyboard", "confidence": 0.35976797342300415, "x1": 9.0, "y1": 211.0, "x2": 105.0, "y2": 272.0}
]
}`
Issue:

Bounding boxes overlap each other.
Bounding boxes are inaccurately positioned.
Bounding box sizes are too small

How can I ensure that the detected objects' bounding boxes are accurately positioned in the real world?


Подробнее здесь: https://stackoverflow.com/questions/786 ... ased-on-2d