Skip to content

Working with the Camera in Unity

Adjust the camera to deliver well-focused camera frames to the Vuforia Engine. Use the Image class returned by the CameraDevice to access camera frames from your application.

Set Modes for Camera, Focus, and Exposure

Adjust the camera-related modes to optimize its performance against changing environmental conditions.

As long as Vuforia Engine has access to the camera, other device apps cannot access it. Release the camera by stopping Vuforia Engine to allow other applications to access it:

1
VuforiaBehaviour.Instance.enabled = false;

In Unity, this is done automatically when the app is being paused.

Camera mode

The CameraMode can be adjusted for default, speed, or quality from the Camera Device Mode in Vuforia Configuration in the Inspector Window of the ARCamera GameObject.

Alternatively, you can configure the camera mode with a script inherited from Monobehaviour. This registers a callback with the VuforiaBehaviour that will set a camera mode when the Vuforia Engine has started.

1
2
3
4
5
6
7
8
9
void Start()
{
    VuforiaApplication.Instance.OnVuforiaStarted += OnVuforiaStarted;
}

private void OnVuforiaStarted()
{
    VuforiaBehaviour.Instance.CameraDevice.SetCameraMode(Vuforia.CameraMode.MODE_DEFAULT);
}
Enumerator for CameraMode
MODE_DEFAULT Best compromise between speed and quality.
MODE_OPTIMIZE_SPEED Minimize Vuforia Engine impact on the system.
MODE_OPTIMIZE_QUALITY Optimize for better image and tracking quality. Applies a higher resource impact on the system.

The MODE_DEFAULT is used if nothing else is set.

Change the mode to MODE_OPTIMIZE_SPEED if the Vuforia Engine impacts the performance of your device. We recommend using the default mode if you expect the target to be moved continuously

Focus mode

The FocusMode is set by the CameraDevice class. We recommend using the continuous auto-focus mode for most scenarios. Not all devices support all focus modes.

1
2
3
4
5
bool focusModeSet = VuforiaBehaviour.Instance.CameraDevice.SetFocusMode(FocusMode.FOCUS_MODE_CONTINUOUSAUTO);
if (!focusModeSet)
   {
        Debug.Log("Failed to set focus mode" + focusModeSet);
   }
Focus mode enum FocusMode Behavior
FOCUS_MODE_FIXED Sets the camera into a fixed focus defined by the camera driver.
FOCUS_MODE_TRIGGERAUTO Triggers a single autofocus operation.
After the operation is completed, the focus mode will automatically change to FOCUS_MODE_FIXED.
Setting a CameraRegionOfInterest requires TRIGGERAUTO to be set afterwards.
FOCUS_MODE_CONTINUOUSAUTO Lets you turn on driver-level continuous autofocus for cameras. This mode is recommended as it guarantees that the camera is always focused on the target.
FOCUS_MODE_INFINITY Sets the camera to infinity, as provided by the camera driver implementation. This mode will set the camera to always focus on the background elements in the scene. (Only supported on UWP and Android without ARCore).
FOCUS_MODE_MACRO Sets the camera to macro mode, as provided by the camera driver implementation. This mode provides a sharp camera image for closeups of approximately 15 cm, rarely used in AR setups. (Not supported on iOS and Magic Leap)

If nothing else is set, the platform's default focus mode is used.

Focus region

Warning

NOTE: The camera controls for setting/getting focus and exposure regions are not supported on devices running ARCore. Android devices with ARCore disabled can use the controls.

Allow your users to focus on just a region of the camera frame. Set the focus region with a CameraRegionOfInterest representing a region-of-interest screen position in pixels and an extent that is a percentage of the size of the camera frame's width and height. This allows the user to tap the screen to focus on a particular screen area.

CameraRegionOfInterest must be followed by setting focus mode to FOCUS_MODE_CONTINUOUSAUTO to trigger the re-focus. After the re-focus, the mode sets itself to FOCUS_MODE_FIXED.

NOTE: Focusing only on a region of the camera view can interfere with and degrade the detection and tracking of Vuforia targets outside that region.

First, ensure that the device supports setting the focus region with FocusRegionSupported while the Engine is running. Then, set the focus region with the regionOfInterest data structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
void Update()
{
    if (Input.touchCount > 0)
    {
        var touch = Input.GetTouch(0);
        if (touch.phase == TouchPhase.Ended)
        {
            SetFocusRegion(touch.position, 0.25f);
        }
    }
}

void SetFocusRegion(Vector2 focusPosition, float extent)
{
    var regionOfInterest = new CameraRegionOfInterest(focusPosition, extent);
    if (VuforiaBehaviour.Instance.CameraDevice.FocusRegionSupported == true)
    {
        var success = VuforiaBehaviour.Instance.CameraDevice.SetFocusRegion(regionOfInterest);
        if (success)
        {
            VuforiaBehaviour.Instance.CameraDevice.SetFocusMode(FocusMode.FOCUS_MODE_TRIGGERAUTO);
        }
        else
        {
            Debug.Log("Failed to set Focus Mode for region " + regionOfInterest.ToString());
        }
    }
    else
    {
        Debug.Log("Focus region supported: " + VuforiaBehaviour.Instance.CameraDevice.FocusRegionSupported);
        VuforiaBehaviour.Instance.CameraDevice.SetFocusRegion(CameraRegionOfInterest.Default());
    }
}

Use screen coordinates in pixels as an alternative to the touch position to provide the focus region position in CameraOfInterest().

Reset the focus region with CameraRegionOfInterest.Default() or set CameraRegionOfInterest to:

1
VuforiaBehaviour.Instance.CameraDevice.SetFocusRegion(new CameraRegionOfInterest(new(Screen.width / 2f, Screen.height / 2f), 1.0f));

If Vuforia Engine is paused, the focus region will be reset and reverted to its default values.

NOTE: On UWP, setting an extent (For focus and exposure) of less than 0.05 (5%) throws an error as the platform deems it too small a value.

Use the get method to return the currently active region for auto-focus.

1
GetFocusRegion();

NOTE: On iOS, only the screen position is returned. The extent (For focus and exposure) is returned as a 0 as iOS changes this value internally. The extent should still be set in the CameraregionOfInterest.

Exposure mode

Adjust the exposure on the camera frames by setting the exposure mode to correct for lighting settings. Note that not all devices support all available exposure modes.

First, verify that the exposure mode you want to change to is supported on the device:

1
2
var exposureMode = ExposureMode.EXPOSURE_MODE_FIXED;
IsExposureModeSupported(exposureMode);

Set the exposure mode while the Vuforia Engine is running with:

1
VuforiaBehaviour.Instance.CameraDevice.SetExposureMode(ExposureMode.EXPOSURE_MODE_TRIGGERAUTO);

The platform's default exposure mode is used if nothing else is set.

Exposure mode ExposureMode
EXPOSURE_MODE_TRIGGERAUTO Triggers a single auto-exposure operation.
After the operation is completed, the exposure mode will automatically change to EXPOSURE_MODE_FIXED.
Setting a CameraRegionOfInterest requires TRIGGERAUTO to be set afterwards.
EXPOSURE_MODE_CONTINUOUSAUTO Allows the device to control auto-exposure continuously. This mode is recommended as it guarantees that the camera is always applying exposure corrections on the camera view.
EXPOSURE_MODE_FIXED Sets the exposure at a fixed value defined by the camera driver.

In most scenarios, the EXPOSURE_MODE_CONTINUOUSAUTO is the recommended mode. Apply other modes when the environment has strong or insufficient light to show the target.

Setting or changing the exposure mode while the Vuforia Engine is running might take a little longer on specific devices until the exposure changes.

Exposure region

Warning

NOTE: The camera controls for setting/getting focus and exposure regions are not supported on devices running ARCore. Android devices with ARCore disabled can use the controls.

Allow your users to set the exposure on just a region of the camera frame with the CameraRegionOfInterest data structure. Set the exposure region with a region-of-interest screen position in pixels and an extent that is a percentage of the size of the camera frame's width and height. CameraRegionOfInterest must be followed by setting exposure mode to EXPOSURE_MODE_TRIGGERAUTO to trigger the re-exposure. After the re-exposure, the mode sets itself to EXPOSURE_MODE_FIXED.

NOTE: Adjusting the exposure on only a region of the camera view can interfere with and degrade the detection and tracking of Vuforia targets outside that region.

First, ensure that the device supports setting the exposure region with ExposureRegionSupported while Vuforia Engine runs. Then, set the exposure region with the regionOfInterest variable:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
void Update()
{
    if (Input.touchCount > 0)
    {
        var touch = Input.GetTouch(0);
        if (touch.phase == TouchPhase.Ended)
        {
            SetFocusRegion(touch.position, 0.25f);
        }
    }
}

void SetExposureRegion(Vector2 focusPosition, float extent)
{
    var regionOfInterest = new CameraRegionOfInterest(focusPosition, extent);
    if (VuforiaBehaviour.Instance.CameraDevice.ExposureRegionSupported == true)
    {
        var success = VuforiaBehaviour.Instance.CameraDevice.SetExposureRegion(regionOfInterest);
        if (success)
        {
            VuforiaBehaviour.Instance.CameraDevice.SetExposureMode(ExposureMode.EXPOSURE_MODE_TRIGGERAUTO);
        }
        else
        {
            Debug.Log("Failed to set Exposure Mode for region " + regionOfInterest.ToString());
        }
    }
    else
    {
        Debug.Log("Exposure region supported: " + VuforiaBehaviour.Instance.CameraDevice.ExposureRegionSupported);
        VuforiaBehaviour.Instance.CameraDevice.SetExposureRegion(CameraRegionOfInterest.Default());
    }
}

Use screen coordinates in pixels as an alternative to the touch position to provide the exposure region position in CameraOfInterest(). Reset the focus region with CameraRegionOfInterest.Default() or set CameraRegionOfInterest to:

1
VuforiaBehaviour.Instance.CameraDevice.SetExposureRegion(new CameraRegionOfInterest(new(Screen.width / 2f, Screen.height / 2f), 1.0f));

NOTE: On iOS, only the screen position is returned. The extent is returned as a 0 as iOS changes this value internally. The extent should still be set in the CameraregionOfInterest.

If Vuforia Engine is paused, the exposure region will be reset and reverted to its default values.

Use the get method to return the currently active region for auto-exposure.

1
GetExposureRegion();

Flash Torch

Poor lighting conditions can significantly affect target detection and tracking. For best results, ensure there is enough light in your environment so that the scene details and target features are well visible in the camera view.

  • Tracking works best in indoor environments, where the lighting conditions are usually more stable and easier to control.

  • If your application use case and scenarios require operating in dark environments, consider enabling the device Flash torch (if your device has one).

    Warning

    NOTE: ARCore version 1.45 or later must be included in the project and installed on the ARCore-enabled Android devices to use the Flash. See Using ARCore with Vuforia Engine in Unity on changing the version.

Call IsFlashSupported() while Vuforia Engine is running to check if the platform device supports setting the Flash mode.

1
VuforiaBehaviour.Instance.CameraDevice.IsFlashSupported();

Set Flash to true to activate or false to deactivate.

1
VuforiaBehaviour.Instance.CameraDevice.SetFlash(true);

Clipping Plane

If your augmentations disappear at a certain distance from the Image Target, it may be that your far clipping plane (in OpenGL or the Unity camera settings) needs to be adjusted. This applies especially if you work with large Image Targets viewed from a distance.

In Unity, the near and far clipping planes can be set directly in the ARCamera GameObject's Inspector window or with Unity's scripting API.

Apply a Shader to the Camera Images

The Image class provides the camera pixels as a byte array. This approach is useful for some image processing tasks, but sometimes, it is preferable to obtain the image as a Unity Texture2D, as shown in the example below. You can also apply the image to a Material's texture in a shader.

Access the Camera Image

Use the Vuforia Image class to access and set the desired camera image format. Use the images as an OpenGL texture or apply the texture to a Unity material. Register for the desired image format using the Vuforia.PixelFormat declaration.

1
VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(PixelFormat.RGB888, true);

NOTE: The Vuforia Namespace Reference page lists the available pixel formats.

Call this method after Vuforia Engine is initialized and started; to this aim, it is recommended to register an OnVuforiaStarted callback in the Start() method of your MonoBehaviour script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
private void OnVuforiaStarted()
{
    PixelFormat pixelFormat = PixelFormat.RGB888;
    bool success = VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(pixelFormat, true);

    // Vuforia has started, now register camera image format
    if (success)
    {
        Debug.Log("Successfully registered pixel format " + pixelFormat.ToString());
    }
    else
    {
        Debug.LogError(
            "Failed to register pixel format " + pixelFormat.ToString() +
            "\n the format may be unsupported by your device;" +
            "\n consider using a different pixel format.");
    }
}

We can extend this script to retrieve the camera images after setting the format and during a callback. In this way, you ensure you obtain the latest camera image matching the current frame.

Also, make sure that the camera image is not null since it can take a few frames for the image to become available after registering for an image format.

Unregister the camera image format whenever the Engine is stopped and register it again when the Engine is started again.

Apply camera images to a RawImage GameObject in your Unity scene.

  1. Create an Empty GameObject and attach the below script CameraImageAccess.
  2. Create a RawImage GameObject from UI -> RawImage.
  3. Drag the RawImage GameObject to the public field in the CameraImageAccess script.
CameraImageAccess.cs
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
using UnityEngine;
using UnityEngine.UI;
using Vuforia;
using Image = Vuforia.Image;

public class CameraImageAccess : MonoBehaviour
{
    const PixelFormat PIXEL_FORMAT = PixelFormat.RGB888;
    const TextureFormat TEXTURE_FORMAT = TextureFormat.RGB24;

    public RawImage RawImage;

    Texture2D mTexture;
    bool mFormatRegistered;

    void Start()
    {
        // Register Vuforia Engine life-cycle callbacks:
        VuforiaApplication.Instance.OnVuforiaStarted += OnVuforiaStarted;
        VuforiaApplication.Instance.OnVuforiaStopped += OnVuforiaStopped;
        if (VuforiaBehaviour.Instance != null)
            VuforiaBehaviour.Instance.World.OnStateUpdated += OnVuforiaUpdated;
    }

    void OnDestroy()
    {
        // Unregister Vuforia Engine life-cycle callbacks:
        if (VuforiaBehaviour.Instance != null)
            VuforiaBehaviour.Instance.World.OnStateUpdated -= OnVuforiaUpdated;

        VuforiaApplication.Instance.OnVuforiaStarted -= OnVuforiaStarted;
        VuforiaApplication.Instance.OnVuforiaStopped -= OnVuforiaStopped;

        if (VuforiaApplication.Instance.IsRunning)
        {
            // If Vuforia Engine is still running, unregister the camera pixel format to avoid unnecessary overhead
            // Formats can only be registered and unregistered while Vuforia Engine is running
            UnregisterFormat();
        }

        if (mTexture != null)
            Destroy(mTexture);
    }

    /// 
    /// Called each time the Vuforia Engine is started
    /// 
    void OnVuforiaStarted()
    {
        mTexture = new Texture2D(0, 0, TEXTURE_FORMAT, false);
        // A format cannot be registered if Vuforia Engine is not running
        RegisterFormat();
    }

    /// 
    /// Called each time the Vuforia Engine is stopped
    /// 
    void OnVuforiaStopped()
    {
        // A format cannot be unregistered after OnVuforiaStopped
        UnregisterFormat();
        if (mTexture != null)
            Destroy(mTexture);
    }

    /// 
    /// Called each time the Vuforia Engine state is updated
    /// 
    void OnVuforiaUpdated()
    {
        var image = VuforiaBehaviour.Instance.CameraDevice.GetCameraImage(PIXEL_FORMAT);

        // There can be a delay of several frames until the camera image becomes available
        if (Image.IsNullOrEmpty(image))
            return;

        Debug.Log("\nImage Format: " + image.PixelFormat +
                  "\nImage Size: " + image.Width + " x " + image.Height +
                  "\nBuffer Size: " + image.BufferWidth + " x " + image.BufferHeight +
                  "\nImage Stride: " + image.Stride + "\n");

        // Override the current texture by copying into it the camera image flipped on the Y axis
        // The texture is resized to match the camera image size
        image.CopyToTexture(mTexture, true);

        RawImage.texture = mTexture;
        RawImage.material.mainTexture = mTexture;
    }

    /// 
    /// Register the camera pixel format
    /// 
    void RegisterFormat()
    {
        // Vuforia Engine has started, now register camera image format
        var success = VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(PIXEL_FORMAT, true);
        if (success)
        {
            Debug.Log("Successfully registered pixel format " + PIXEL_FORMAT);
            mFormatRegistered = true;
        }
        else
        {
            Debug.LogError("Failed to register pixel format " + PIXEL_FORMAT +
                           "\n the format may be unsupported by your device;" +
                           "\n consider using a different pixel format.");
            mFormatRegistered = false;
        }
    }

    /// 
    /// Unregister the camera pixel format
    /// 
    void UnregisterFormat()
    {
        Debug.Log("Unregistering camera pixel format " + PIXEL_FORMAT);
        VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(PIXEL_FORMAT, false);
        mFormatRegistered = false;
    }
}

Use an OpenGL texture

Other image processing tasks might require obtaining the image as an OpenGL texture. You can get the image as an OpenGL texture using the approach demonstrated in the OcclusionManagement sample.

  1. Register a Texture2D object to be filled with the camera pixels at each frame instead of letting Vuforia Engine render the camera image natively at each frame, using the VuforiaRenderer.VideoBackgroundTexture API
  2. See the OcclusionManagement sample scripts for an example of this technique.

Accessing the Raw Pixels

For custom processes involving CV (Computer Vision), you can access the raw pixels from the camera with the Image class. See the following example method for retrieving the pixel size:

1
2
3
4
5
6
7
8
private byte[] GetPixels(Image image)
{
    var pixelBufferPtr = image.PixelBufferPtr;
    int imageSize = image.Stride * image.Height;
    byte[] pixels = new byte[imageSize];
    System.Runtime.InteropServices.Marshal.Copy(pixelBufferPtr, pixels, 0, imageSize);
    return pixels;
}

Learn More

Device Performance Optimization