Step Check for Vuforia Engine - Beta

Step Check visually verifies whether a step in a procedure has been performed correctly. This feature works without needing to upload images or training data to Vuforia cloud services. Use a single Template image to help users verify if a step was performed correctly.

Info

Interested? Reach out to vuforia-feedback@ptc.com to sign up for the beta.

Introduction

Step Check for Vuforia Engine is a powerful technology that validates whether a step has been performed correctly from a single image. It allows you to amplify your instructions with consistent checks on assembly instructions, visual inspections, manuals, and interactive experiences. Use this feature to enhance workers with visual validation tools powered by Vuforia Engine.

Use cases include end-of-line inspections, verifying that a step in a series of assembly instructions was done correctly, and validation of service instructions.

Prerequisites

An Enterprise license is required to test and use this feature.

A platform-dependent plugin file is needed to run the Step Check feature.

Device support

• iPhone 12+:
  • Requires iOS 18+.
• iPad support:
  • iPad Pro 5th Gen and later.
  • iPad 10th Gen and later.
  • iPad Air 4th Gen and later.
  • iOS 18+.
• Android support - recommended devices:
  • Samsung Galaxy S24 and S25 family of devices.
  • Google Pixel 8 and 9 family of devices.

Workstation support

• Testing using Unity Play Mode:
  • Mac: MacOS 15.0 or later with an M1 or later chipset.
  • Windows

Overview

The Step Check feature is configured by a masked template image consisting of:

• A color image of the validation area and its surroundings.
• A mask that defines what part of the image shall be validated (stored in the alpha channel of the image).
• A label that will be returned if the validation area was validated successfully.

In the example below, Step Check is configured to validate that the oil dipstick of a car engine is fully inserted (e.g. after doing an oil check). On successful validation, a pass label will be returned. The label can be freely defined.

Validation workflow:

• Once Step Check has recognized the area from the image in the camera view, a validation result will be returned.
• Step Check will validate if the area defined by the mask matches what is seen by the camera.
• If validation was successful, validation status NORMAL and the label of the recognized template will be returned.
• Otherwise, validation status UNDECIDABLE will be returned.

Best Practices

Follow the instructions below to ensure best Step Check performance.

Image Content

The image should be representative of the camera view when the validation should be performed. Therefore, it should be taken from a similar position and distance that a worker will be in when performing the inspection or service procedure. It should not be cropped as this will change the distance from which the area can be recognized.

To some degree, Step Check is robust against changes in lighting and background, but ideally the image should mostly consist of the object that is being serviced or assembled. Avoid including a highly textured background in the image that might look different during the actual validation.

The image and, more precisely, the validation area, should be taken of the state that needs to be validated, that is, the correct setup, installation, fit, or removal of a part.

Image Quality

• The lighting on the Step Check image should be as close as possible to when performing the validation check. Changing lighting conditions influence how well the template image can be recognized. Stable lighting on an indoor factory floor would be sufficient. In contrast, an outdoor setting with changing direct natural light might cause recognition issues.
• Capture the image so that the validation area is in the center of the image. This practice will avoid accidentally cropping out the validation area and ensure the utilization of the surroundings on all sides.
• Take the image so that the validation area consists of a minimum of 10% of the image.
• Avoid zooming in with the camera as it will decrease the image quality. On a mobile device, zoom level 1 should be used to take the image.

Mask Definition

The alpha layer mask defines the area that is validated:

• Keep the mask as tightly to the part to validate as possible. Avoid including parts or areas that might look different for every instance, but are not important to be validated (e.g., serial numbers).
• Avoid reflections, dirt etc. within the mask that won't be seen during validation or will look different every time.
• A template should only mask a single validation area, not multiple parts that need to be independently checked. For instance, if a part is fastened with 4 different screws, you should avoid masking all 4 screws in a single template. This check would only pass successfully if all 4 screws are visible and in the correct state at the same time.
  • Instead, 4 different template images should be defined with a mask for every single screw. Multiple observers can be created at the same time to perform those 4 step checks in parallel, but independently.

Example: Power Switch

In this example, we want to validate that a device is switched off, e.g. before performing a maintenance procedure. In the template image, the masked switch is in the OFF position. If Step Check is recognizing this view and the switch is ON, the contents of the mask will not match and validation will fail.

Example: Connector

In this example, we want to validate that a connector is correctly engaged and fully screwed in. Validation will only succeed if the connector is fully visible in the mask. If it is missing or not fully screwed in, validation will fail.

Example: Valve

In this example, we want to validate that the valve on a water pipe is opened. Both handles of the valve are masked. Validation will only succeed if both handles are in the masked areas. If the valve is closed or not fully open, validation will fail.

Example: Audio jack

In this example, we want to validate that the audio jack connector is fully inserted, while the rotation of the connector is irrelevant. Validation will in this scenario only succeed if the audio connector is fully inserted. If the connector is not fully inserted, validation will fail.

Set up a Template Image

Use the Vuforia Creator App version 11.1 or later to capture template images and mask validation areas. Alternatively, images can be prepared manually using common image-editing software; see Advanced Topics.

In the Creator App, select Create Step Check Template from the + menu in the Asset Browser

Take a photo of the area you want to validate from roughly the distance and viewpoint from which users would be. Review the photo and select DONE to confirm.

You can now create a mask by drawing with your finger on the screen. The image will be displayed in greyscale, with the mask highlighted in bright green.

Use the following tools for drawing:

• Two fingers to zoom - the brush size will stay constant; zoom in to draw finer details.
• Three fingers to move the image while zoomed.
• The "eye" icon will toggle between color image and greyscale + mask.
• The green icon will enable drawing.
• The eraser icon will enable mask erasing.

When finished, save the image and enter a name.

The template image is now ready to be tested using the Vuforia Creator App (see next section) or used in a Vuforia Engine app. To share it through Dropbox, OneDrive, Email, etc., long-press the image in the asset browser and select Share. You can also re-open and edit the masked validation area of an existing template image.

NOTE: Instead of taking a photo using the camera, you can also import an existing image. It needs to be a PNG image with an alpha channel. You can then open and edit it to change or create a new mask.

Test Step Check in the Vuforia Creator App

Use the Vuforia Creator App version 11.0 or later to quickly and easily test Step Check template images on supported devices.

First, import the onprem-stepcheck-apple-plugin-v1.arc file and the Step Check template image. The PNG file includes the masked validation area in the alpha channel. If you created the template image with the Creator app, it will already be listed in your assets.

To select the imported Template image, you must first be logged in with your Vuforia Engine developer account, which must also be enrolled in the Step Check Beta to test this feature. Once authenticated, selecting a Step Check Template image will start an AR experience. The UI will display the template image in the lower left corner to guide the user where to look. The reticle in the center will show the validation result and confidence value.

If Vuforia Engine finds a 3D pose, it will display the result as an icon augmentation at the location of the validation area; if it does not find a pose, the Creator App will show the result as a 2D UI next to the center reticle. The Creator App always visualizes a successful validation with a green pass icon.

Once a 3D pose is available, the app will display a cone to guide the user to the best viewpoint to validate. The viewpoint is the position from which you captured the template image. If the validation area is out of camera view, a sling will be displayed.

Unity Sample

Try out Step Check in our Unity sample project. Currently, the sample project is not available in the Asset Store and only distributed to the developers participating in the beta.

1. Start a new Unity Project and import the sample Unity Asset Package from Assets -> Import Package -> Custom Package...

2. Enter your Enterprise App license key in Window -> Vuforia Configuration -> App License Key

3. Explore the sample: it consists of four scenes with two test cases that can be found in Assets/SampleResources/Scenes:

   1. 0-Splash: Splash screen scene.

   2. 1-Info: Info menu with explanation and access to sample scenes.

   3. 2-OilDipstick: Sample scene showing how to use Step Check to check an oil dipstick using a single template to validate the correct pass state.

   4. 2-PowerSwitch: Slightly more complex scene showing how to use Step Check to check if a power switch is on or off. This scene uses two ValidationArea GameObjects, one configured with a pass template, and another with a fail template showing the incorrect state.

Session Recordings are included and can be played on a Mac in play mode in the Unity Editor.

NOTE: When using Play Mode in Unity, the icons will appear at the top right of the screen. On a mobile device, they will appear over the validation area. This is because when using the webcam or Session Recordings, the 3D pose information to position the icons is missing.

To test your own Step Check template, you can get started quickly by duplicating one of the existing sample scenes and modifying the ValidationArea GameObject with your own Template image.

1. First, import your own Template image, e.g. next to existing templates in Assets/SampleResources/StepCheckTemplates
2. Import the original image as well; it can be used to guide users to the validation area.

3. Select the ValidationArea GameObject and replace the Template Image.

   NOTE: Depending on how the image was set up, there may be some errors or warnings in the inspector. To fix them, click the Apply changes to Texture-Import Properties button.

4. In the Hierarchy, navigate to the child GameObject Preview image and replace it with the original image you imported earlier.

5. Optionally, update any of the other content, such as the icons and title.

From here, you can add additional ValidationArea GameObjects to the scene and more sophisticated instructions to create a complete work procedure experience.

Unity Extension

Set up a new scene

These steps are for setting up an empty scene without the Unity sample. The Vuforia SDK Unity Extension 11.0 or later is required for this.

1. Import the latest Vuforia Engine SDK for Unity.
2. Right-click in the Hierarchy, and add an ARCamera, and a Validation Area (Beta) under Step Check.
3. Import your Template image into the project.
4. In the ValidationArea GameObject, fill in the Template image, Target Name, and Template Label fields. You can also add an Icon Behaviour once the Template image has been populated.
5. Add an Enterprise license to the Vuforia Configuration before building to the device or testing in Play Mode.
6. Import the onprem-stepcheck-apple-plugin-v1.arc and onprem-stepcheck-android-win-plugin-v1.arc files into the Assets/StreamingAssets/Vuforia folder.

Unity Runtime API

Create a ValidationAreaBehaviour at runtime with ObserverFactory.CreateValidationArea or ObserverFactory.CreateValidationAreaAsync. The method requires a label, validation area name, and an input Template image that can be either from a file path, pixel buffer, or a Texture2D. The image requires that you already have masked an area following the guidelines in the Set Up Template Image and is of the supported media format RGBA-PNG.

Example code sample for creating a validation area from code:

using UnityEngine;
using Vuforia;

public class RuntimeStepCheckLoader : MonoBehaviour
{
    public string filePath;
    public string label;
    public string validationAreaName;    

    // Start is called before the first frame update
    void Start()
    {
        ValidationAreaBehaviour vaBehaviour = VuforiaBehaviour.Instance.ObserverFactory.CreateValidationArea(
            filePath,
            label,
            validationAreaName
        );

        vaBehaviour.OnValidationInfoUpdateReceived += ValidationInfoUpdated;
    }

    void ValidationInfoUpdated(ValidationAreaBehaviour behaviour, ValidationInfo validationInfo)
    {
        // Parse ValidationInfo result
    }
}

Subscribe to the OnValidationInfoUpdateReceived event to receive updates whenever the ValidationInfo changes. It contains the following information:

• ValidationStatus (NORMAL or UNDECIDABLE)
• Label
• Confidence value
• Guidance to the optimal viewpoint for validation. This should only be used if the Validation Area has been recognized and is being tracked (ValidationAreaBehaviour.TargetStatus.Status != Status.NO_POSE). See the Unity API Reference for more information. Currently, only iOS supports guidance.

See how ValidationInfoUpdated is set up in the ValidationAreaController.cs in the Step Check Unity sample.

You can also query the size of the validation area using the AreaSize property. Note that it currently always returns 3 x 3 x 1 in centimeters.

By default, the Step Check plugin files needs to be placed in Assets/StreamingAssets/Vuforia. In case another folder is preferred, it needs to be set through VuforiaConfiguration.Instance.Plugins.PluginDirectory before initializing Vuforia Engine.

Native API Overview

Create a Validation Area Observer using a file path to a template image or from a pixel buffer. Note that images with a height or width bigger than 2048 pixels are note supported. Images need to be of RGBA-PNG format.

An Enterprise license is required for Step Check. Add the key to constexpr char licenseKey[] = "";.

VuValidationAreaFileConfig config = vuValidationAreaFileConfigDefault();
config.devicePoseObserver = devicePoseObserver;
config.path = "filePath";
config.label = "pass";
config.validationAreaName = "validation_area_subject";
config.activate = VU_TRUE;

VuValidationAreaFileCreationError validationAreaErrorCode;
VuObserver* validationAreaObserver = NULL;;

vuEngineCreateValidationAreaObserverFromFileConfig(engine, &validationAreaObserver, config, validationAreaErrorCode);

Similar to other Observers, the observations can be retrieved with the following:

/ Create observation list
VuObservationList* obsList = nullptr;
vuObservationListCreate(&obsList);

// Get and parse validation area observations list
vuStateGetValidationAreaObservations(state, obsList);

int32_t listSize = 0;
vuObservationListGetSize(obsList, &listSize);

And parsing the list to get info on the validation area:

// Parse the validation area list
for (int i = 0; i < listSize; i++)
{
    VuObservation* obs = nullptr;
    vuObservationListGetElement(obsList, i, &obs);

    VuPoseInfo poseInfo;
    vuObservationGetPoseInfo(obs, &poseInfo);

    VuValidationAreaObservationStatusInfo statusInfo;
    vuValidationAreaObservationGetStatusInfo(obs, &statusInfo);

    if (validationInfo.validationStatus != VU_VALIDATION_AREA_OBSERVATION_STATUS_INFO_NOT_OBSERVED)
    {
        // Do something with poseInfo and statusInfo
    }
}

Get the ValidationInfo to access the validation status, label, confidence metric, and recommendations for viewer direction, angle, and distance. The ValidationInfo reporting is independent of the pose estimation, which means there might be validation results even when there is no pose. See the C API Reference for more information.

VuValidationAreaObservationValidationInfo validationInfo;
vuValidationAreaObservationGetValidationInfo(obs, &validationInfo);

Plugin directory

To use Step Check, you are required to load a path to the plugin files during the Vuforia Engine initialization. The plugin files are then loaded during the ValidationAreaObserver creation.

The plugin files must be placed in a common plugin folder that can be configured in the Vuforia Engine config during initialization.

VuEngineConfigSet* configSet = nullptr;
VuResult vuResult = vuEngineConfigSetCreate(&configSet);
assert(vuResult == VU_SUCCESS);
VuPluginConfig pluginConfig = vuPluginConfigDefault();
pluginConfig.pluginDirectory = "<PATH_TO_COMMON_FOLDER>";

vuEngineConfigSetAddPluginConfig(configSet, &pluginConfig);

Learn More

Advanced Step Check Topics

Advanced Model Targets

On-prem Advanced Model Targets

Model Targets in Unity