Mesh Observer API Overview

The Mesh Observer provides runtime meshes based on Area Targets, Area Target Captures, and Model Targets. Create meshes to add occlusion for your tracking objects and environments and use the meshes for visualizations of your targets.

Prerequisites

The Mesh Observer requires an Area Target Observer, an Area Target Capture instance, or a Model Target Observer as its source.

Create Mesh Observer from Area Target

Create the Mesh Observer with a VuMeshAreaTargetConfig that associates the Mesh Observer with an Area Target Observer and its occlusion mesh. Thereafter, retrieve the Mesh Observations to render the result. See the Render Mesh at Runtime for details and example snippet.

The Mesh Observer configuration VuMeshAreaTargetConfig requires an Area Target Observer, an external path to the Area Target’s occlusion .3dt file, and the parameter activate that can be set to VU_FALSE or VU_TRUE.

12345678910111213141516
Copy
// Create the Mesh Observer configuration from Area Target source VuMeshAreaTargetConfig meshObserverConfig = vuMeshAreaTargetConfigDefautlt(); // Must be a valid Area Target Observer meshObserverConfig.areaTargetObserver = areaTargetObserver; // Must be a valid path to an existing occlusion 3dt file meshObserverConfig.occlusionMeshPath = "pathAsString"; // Auto-activation on creation is the default meshObserverConfig.activate = VU_TRUE; // Create the Mesh Observer VuObserver* meshObserver = nullptr; VuMeshAreaTargetCreationError error = VU_MESH_AREA_TARGET_CREATION_ERROR_NONE; vuEngineCreateMeshObserverFromAreaTargetConfig(engine, &meshObserver, &meshObserverConfig, &error);

The Area Target Observer is the exclusive source of the Mesh Observations reported by the Mesh Observer. The Mesh Observer aligns its Mesh Observations with respect to the areaTargetObserver.

The occlusionMeshPath is the external path to the occlusion mesh artifact (occlusion.3dt). The occlusion mesh must originate from the Area Target that is being tracked by areaTargetObserver.

Destroy the Mesh Observer.

1
Copy
vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Area Target Observer from which it is drawing the Mesh observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Observer.

Create Mesh Observer from Area Target Capture

Create the Mesh Observer and retrieve live mesh updates during capturing from an Area Target Capture instance and provide users helpful UX feedback on the scanned areas during capturing. The Mesh Observer configuration requires an Area Target Capture instance and the parameter activate that can be set to VU_FALSE or VU_TRUE.

12345678910111213
Copy
// Create the Mesh Observer configuration from Area Target Capture source VuMeshAreaTargetCaptureConfig config = vuMeshAreaTargetCaptureConfigDefault(); // Must be a valid VuAreaTargetCapture instance config.capture = capture; // Auto-activation on creation is the default config.activate= VU_TRUE; // Create the Mesh Observer VuObserver* meshObserver = nullptr; VuMeshAreaTargetCaptureCreationError error = VU_MESH_AREA_TARGET_CAPTURE_CREATION_ERROR_NONE; vuEngineCreateMeshObserverFromAreaTargetCaptureConfig(engine, meshObserver, &config, &error);

The Area Target Capture instance is the exclusive source of the Mesh Observations reported by the Mesh Observer.

Destroy the Mesh Observer.

1
Copy
vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Area Target Capture instance from which it is drawing the Mesh Observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Capture instance.

Create Mesh Observer from Model Target

Create the Mesh Observer with a VuMeshModelTargetConfig that associates the Mesh Observer with a Model Target Observer. Thereafter, retrieve the Mesh Observations to render the result. See the Mesh Observations and Render Mesh at Runtime for details and example snippet.

12345678910111213
Copy
// Create the Mesh Observer configuration from a Model Target source VuMeshModelTargetConfig meshObserverConfig = vuMeshModelTargetConfigDefault(); // Must be a valid Model Target Observer meshObserverConfig.modelTargetObserver = modelTargetObserver; // Auto-activation on creation is the default meshObserverConfig.activate = VU_TRUE; // Create the Mesh Observer VuObserver* meshObserver = nullptr; VuMeshModelTargetCreationError error = VU_MESH_MODEL_TARGET_CREATION_ERROR_NONE; vuEngineCreateMeshObserverFromModelTargetConfig(engine, &meshObserver, &meshObserverConfig, &error);

The Model Target Observer is the exclusive source of the Mesh Observations reported by the Mesh Observer. The Mesh Observer aligns its Mesh Observations with respect to the Model Target Observer.

Destroy the Mesh Observer after usage.

1
Copy
vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Model Target Observer from which it is drawing the Mesh observations from. Therefore, destroy the Mesh Observer before you destroy the Model Target Observer.

Mesh Observations

Retrieve the Mesh Observations from the Engine State. Create an observation list to hold the Mesh Observations. Retrieving the observation information from the state is the same for the Mesh Observer from an Area Target and from an Area Target Capture, and Model Target source.

12345678910
Copy
// Create observation list VuObservationList* observationList = nullptr; vuObservationListCreate(&observationList); // Get mesh observation list int numObservations = 0; vuStateGetObservationsByObserver(state, meshObserver, observationList); // Get mesh observation list size vuObservationListGetSize(observationList, &numObservations);

Each Mesh Observation holds a pose of type VuPoseInfo, a respective status info of type VuMeshObservationStatusInfo, and an object of type VuMeshObservationInfo that holds a list of meshes of type VuMeshObservationBlock.

Mesh Observation Status and Status Info

Each Mesh Observation holds a VuPoseInfo object to align the content of the observation with the world coordinate system. Additionally, each Mesh Observation holds a respective VuMeshObservationStatusInfo to provide more detailed information about the pose.

The reported pose status and status info pairs are shown in the table below.

STATUS

STATUS_INFO

NO_POSE

VU_MESH_OBSERVATION_STATUS_INFO_NOT_OBSERVED

EXTENDED_TRACKED

VU_MESH_OBSERVATION_STATUS_INFO_NORMAL

LIMITED

VU_MESH_OBSERVATION_STATUS_INFO_RELOCALIZING

A Mesh Observer with an Area Target source reports a pose status that corresponds to the pose status of the Area Target Observer.

A Mesh Observer with an Area Target Capture source reports either NO_POSE or a pose with status EXTENDED_TRACKED, but never a pose with status LIMITED.

A Mesh Observer with a Model Target source that has not yet been observed, will publish the last known mesh with an identity pose and a NO_POSE status until the associated source Observer is observed. When observed, the pose status and status info correspond to that of the Model Target Observer.

Mesh Observation Info

Each Mesh Observation holds an object of type VuMeshObservationInfo that holds a list of meshes of type VuMeshObservationBlock.

123
Copy
// Get observation info from a mesh observation VuMeshObservationInfo info; vuMeshObservationGetInfo(meshObservation, &info);

The Mesh Observation info always holds a complete list of meshes. It includes meshes that have been updated, recently added, and meshes that have remained unchanged. Meshes removed from the list will not have their ID reported in the latest retrieved observation.

Mesh observation block

Each VuMeshObservationBlock holds metadata and the actual mesh data. The metadata consists of a unique ID, a timestamp, and a version number. These fields enable the unique identification of each mesh block and allow for tracking of updates to the mesh block.

The VuMeshObservationBlock ID and version are positive numbers. The IDs are unique within a Vuforia session. They are generated at runtime and are not persistent across Vuforia sessions. Moreover, the IDs are not reused, meaning that a newly added mesh block can never have the same ID as an already removed mesh block. Whenever a mesh block is updated, its timestamp is adjusted to reflect the time of the change, and its version is increased.

The actual data of a mesh block consists of a transformation matrix, the axis aligned bounding box of the mesh, and a pointer to a VuMesh structure that holds the mesh. The transform holds the transformation matrix that aligns the block with the observation coordinate system. The Mesh Observation, on the other hand, holds a pose to align the observation with the world coordinate system. The snippet below demonstrates how to assemble a matrix that transforms a mesh block to the world coordinate system.

123
Copy
VuPoseInfo poseInfo; vuObservationGetPoseInfo(meshObservation, &poseInfo); VuMatrix44F modelMatrix = vuMatrix44FMultiplyMatrix(poseInfo.pose, block.transform);

NOTE: A VuPoseInfo with status NO_POSE does not hold a pose to align the Mesh Observation with the world coordinate system

Render Mesh at Runtime

Below you find a simple code snippet that iterates over all Mesh Observations reported by a Mesh Observer and renders the mesh blocks of each Mesh Observation that has a valid pose.

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152
Copy
// Create list that holds the mesh observations VuObservationList *observationList = nullptr; vuObservationListCreate(&observationList); // Retrieve mesh observations from the Mesh Observer vuStateGetObservationsByObserver(&state, meshObserver, observationList); // Retrieve number of mesh observations held by the list int numObservations = 0; vuObservationListGetSize(observationList, &numObservations); // Iterate over all mesh observations for (int o = 0; o < numObservations; o++) { // Retrieve the observation VuObservation *observation = nullptr; vuObservationListGetElement(observationList, o, &observation)); // Retrieve the pose info of the observation VuPoseInfo poseInfo; vuObservationGetPoseInfo(observation, &poseInfo); // Skip if the observation does not have a pose if (poseInfo.poseStatus == VU_OBSERVATION_POSE_STATUS_NO_POSE) { continue; } // Retrieve mesh observation info that holds the list of mesh blocks VuMeshObservationInfo info; vuMeshObservationGetInfo(observation, &info)); // Retrieve number of mesh blocks int32_t numBlocks = 0; vuMeshObservationBlockListGetSize(info.meshes, &numBlocks)); // Iterate over all mesh blocks for (int b = 0; b < numBlocks; b++) { VuMeshObservationBlock block; vuMeshObservationBlockListGetElement(info.meshes, b, &block); // Assemble matrix to transform mesh block to world coordinates VuMatrix44F modelMatrix = vuMatrix44FMultiplyMatrix(poseInfo.pose, block.transform); // Render mesh block render(modelMatrix, block.mesh); } } // Destroy mesh observation list vuObservationListDestroy(observationList);

 

Can this page be better?
Share your feedback via our issue tracker