Mesh Observer API Overview

The Mesh Observer provides runtime meshes based on Area Targets, Area Target Captures, and Model Targets. Create meshes to add occlusion for your tracking objects and environments and use the meshes for visualizations of your targets.

Prerequisites

The Mesh Observer requires an Area Target Observer, an Area Target Capture instance, or a Model Target Observer as its source.

• See Area Target API Overview on how to create an Area Target Observer to use for the Mesh Observer. 
• See Area Target Capture for Native on how to create an Area Target Capture instance to use for the Mesh Observer. 
• See Model Target API Overview on how to create a Model Target Observer to use for the Mesh Observer.

Create Mesh Observer from Area Target

Create the Mesh Observer with a VuMeshAreaTargetConfig that associates the Mesh Observer with an Area Target Observer and its occlusion mesh. Thereafter, retrieve the Mesh Observations to render the result. See the Render Mesh at Runtime for details and example snippet.

The Mesh Observer configuration VuMeshAreaTargetConfig requires an Area Target Observer, an external path to the Area Target's occlusion .3dt file, and the parameter activate that can be set to VU_FALSE or VU_TRUE.

// Create the Mesh Observer configuration from Area Target source
VuMeshAreaTargetConfig meshObserverConfig = vuMeshAreaTargetConfigDefautlt();

// Must be a valid Area Target Observer
meshObserverConfig.areaTargetObserver = areaTargetObserver;

// Must be a valid path to an existing occlusion 3dt file
meshObserverConfig.occlusionMeshPath = "pathAsString";

// Auto-activation on creation is the default
meshObserverConfig.activate = VU_TRUE;

// Create the Mesh Observer 
VuObserver* meshObserver = nullptr;
VuMeshAreaTargetCreationError error = VU_MESH_AREA_TARGET_CREATION_ERROR_NONE;
vuEngineCreateMeshObserverFromAreaTargetConfig(engine, &meshObserver, &meshObserverConfig, &error);

The Area Target Observer is the exclusive source of the Mesh Observations reported by the Mesh Observer. The Mesh Observer aligns its Mesh Observations with respect to the areaTargetObserver.

The occlusionMeshPath is the external path to the occlusion mesh artifact (occlusion.3dt). The occlusion mesh must originate from the Area Target that is being tracked by areaTargetObserver.

Destroy the Mesh Observer.

vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Area Target Observer from which it is drawing the Mesh observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Observer.

Create Mesh Observer from Area Target Capture

Create the Mesh Observer and retrieve live mesh updates during capturing from an Area Target Capture instance and provide users helpful UX feedback on the scanned areas during capturing. The Mesh Observer configuration requires an Area Target Capture instance and the parameter activate that can be set to VU_FALSE or VU_TRUE.

// Create the Mesh Observer configuration from Area Target Capture source
VuMeshAreaTargetCaptureConfig config = vuMeshAreaTargetCaptureConfigDefault();

// Must be a valid VuAreaTargetCapture instance
config.capture = capture; 

// Auto-activation on creation is the default 
config.activate= VU_TRUE;

// Create the Mesh Observer 
VuObserver* meshObserver = nullptr;
VuMeshAreaTargetCaptureCreationError error = VU_MESH_AREA_TARGET_CAPTURE_CREATION_ERROR_NONE;
vuEngineCreateMeshObserverFromAreaTargetCaptureConfig(engine, meshObserver, &config, &error);

The Area Target Capture instance is the exclusive source of the Mesh Observations reported by the Mesh Observer.

Destroy the Mesh Observer.

vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Area Target Capture instance from which it is drawing the Mesh Observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Capture instance.

Create Mesh Observer from Model Target

Create the Mesh Observer with a VuMeshModelTargetConfig that associates the Mesh Observer with a Model Target Observer. Thereafter, retrieve the Mesh Observations to render the result. See the Mesh Observations and Render Mesh at Runtime for details and example snippet.

// Create the Mesh Observer configuration from a Model Target source
VuMeshModelTargetConfig meshObserverConfig = vuMeshModelTargetConfigDefault();

// Must be a valid Model Target Observer
meshObserverConfig.modelTargetObserver = modelTargetObserver;

// Auto-activation on creation is the default
meshObserverConfig.activate = VU_TRUE;

// Create the Mesh Observer 
VuObserver* meshObserver = nullptr;
VuMeshModelTargetCreationError error = VU_MESH_MODEL_TARGET_CREATION_ERROR_NONE;
vuEngineCreateMeshObserverFromModelTargetConfig(engine, &meshObserver, &meshObserverConfig, &error);

The Model Target Observer is the exclusive source of the Mesh Observations reported by the Mesh Observer. The Mesh Observer aligns its Mesh Observations with respect to the Model Target Observer.

Destroy the Mesh Observer after usage.

vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Model Target Observer from which it is drawing the Mesh observations from. Therefore, destroy the Mesh Observer before you destroy the Model Target Observer.

Mesh Observations

Retrieve the Mesh Observations from the Engine State. Create an observation list to hold the Mesh Observations. Retrieving the observation information from the state is the same for the Mesh Observer from an Area Target and from an Area Target Capture, and Model Target source.

// Create observation list
VuObservationList* observationList = nullptr;
vuObservationListCreate(&observationList);

// Get mesh observation list
int numObservations = 0;
vuStateGetObservationsByObserver(state, meshObserver, observationList);

// Get mesh observation list size
vuObservationListGetSize(observationList, &numObservations);

Each Mesh Observation holds a pose of type VuPoseInfo, a respective status info of type VuMeshObservationStatusInfo, and an object of type VuMeshObservationInfo that holds a list of meshes of type VuMeshObservationBlock.

Mesh Observation Status and Status Info

Each Mesh Observation holds a VuPoseInfo object to align the content of the observation with the world coordinate system. Additionally, each Mesh Observation holds a respective VuMeshObservationStatusInfo to provide more detailed information about the pose.

The reported pose status and status info pairs are shown in the table below.

STATUS	STATUS_INFO
NO_POSE	VU_MESH_OBSERVATION_STATUS_INFO_NOT_OBSERVED
EXTENDED_TRACKED	VU_MESH_OBSERVATION_STATUS_INFO_NORMAL
LIMITED	VU_MESH_OBSERVATION_STATUS_INFO_RELOCALIZING

A Mesh Observer with an Area Target source reports a pose status that corresponds to the pose status of the Area Target Observer.

A Mesh Observer with an Area Target Capture source reports either NO_POSE or a pose with status EXTENDED_TRACKED, but never a pose with status LIMITED.

A Mesh Observer with a Model Target source that has not yet been observed, will publish the last known mesh with an identity pose and a NO_POSE status until the associated source Observer is observed. When observed, the pose status and status info correspond to that of the Model Target Observer.

Mesh Observation Info

Each Mesh Observation holds an object of type VuMeshObservationInfo that holds a list of meshes of type VuMeshObservationBlock.

// Get observation info from a mesh observation
VuMeshObservationInfo info;
vuMeshObservationGetInfo(meshObservation, &info);

The Mesh Observation info always holds a complete list of meshes. It includes meshes that have been updated, recently added, and meshes that have remained unchanged. Meshes removed from the list will not have their ID reported in the latest retrieved observation.

Mesh observation block

Each VuMeshObservationBlock holds metadata and the actual mesh data. The metadata consists of a unique ID, a timestamp, and a version number. These fields enable the unique identification of each mesh block and allow for tracking of updates to the mesh block.

The VuMeshObservationBlock ID and version are positive numbers. The IDs are unique within a Vuforia session. They are generated at runtime and are not persistent across Vuforia sessions. Moreover, the IDs are not reused, meaning that a newly added mesh block can never have the same ID as an already removed mesh block. Whenever a mesh block is updated, its timestamp is adjusted to reflect the time of the change, and its version is increased.

The actual data of a mesh block consists of a transformation matrix, the axis aligned bounding box of the mesh, and a pointer to a VuMesh structure that holds the mesh. The transform holds the transformation matrix that aligns the block with the observation coordinate system. The Mesh Observation, on the other hand, holds a pose to align the observation with the world coordinate system. The snippet below demonstrates how to assemble a matrix that transforms a mesh block to the world coordinate system.

VuPoseInfo poseInfo;
vuObservationGetPoseInfo(meshObservation, &poseInfo);
VuMatrix44F modelMatrix = vuMatrix44FMultiplyMatrix(poseInfo.pose, block.transform);

NOTE: A VuPoseInfo with status NO_POSE does not hold a pose to align the Mesh Observation with the world coordinate system

Render Mesh at Runtime

Below you find a simple code snippet that iterates over all Mesh Observations reported by a Mesh Observer and renders the mesh blocks of each Mesh Observation that has a valid pose.

// Create list that holds the mesh observations
VuObservationList *observationList = nullptr;
vuObservationListCreate(&observationList);

// Retrieve mesh observations from the Mesh Observer
vuStateGetObservationsByObserver(&state, meshObserver, observationList);

// Retrieve number of mesh observations held by the list
int numObservations = 0;
vuObservationListGetSize(observationList, &numObservations);

// Iterate over all mesh observations
for (int o = 0; o < numObservations; o++)
{
    // Retrieve the observation
    VuObservation *observation = nullptr;
    vuObservationListGetElement(observationList, o, &observation);

    // Retrieve the pose info of the observation
    VuPoseInfo poseInfo;
    vuObservationGetPoseInfo(observation, &poseInfo);

    // Skip if the observation does not have a pose
    if (poseInfo.poseStatus == VU_OBSERVATION_POSE_STATUS_NO_POSE)
    {
        continue;
    }

    // Retrieve mesh observation info that holds the list of mesh blocks
    VuMeshObservationInfo info;
    vuMeshObservationGetInfo(observation, &info);

    // Retrieve number of mesh blocks
    int32_t numBlocks = 0;
    vuMeshObservationBlockListGetSize(info.meshes, &numBlocks);

    // Iterate over all mesh blocks
    for (int b = 0; b < numBlocks; b++)
    {
        VuMeshObservationBlock block;
        vuMeshObservationBlockListGetElement(info.meshes, b, &block);

        // Assemble matrix to transform mesh block to world coordinates
        VuMatrix44F modelMatrix = vuMatrix44FMultiplyMatrix(poseInfo.pose, block.transform);

        // Render mesh block
        render(modelMatrix, block.mesh);
    }
}

// Destroy mesh observation list
vuObservationListDestroy(observationList);

Learn More

Model Target API Overview

Area Target API Overview

Area Target Capture

Area Target Capture for Native