Vuforia Engine allows developers to record and use video recordings of AR sessions to test their projects. Camera frames and device tracking data are recorded to play back the AR session, enabling developers to rapidly iterate on their app and test AR experiences efficiently. This is particularly helpful for building AR experiences that use Model Targets or Area Targets, where the physical target is not always available for testing. Session Recordings can then be played back on a device or using Unity Editor Play Mode.
Overview and Recommended Workflow
Vuforia Engine includes a SessionRecorder API that can be used directly in a native or Unity app to capture Session Recordings and transfer them from any application using Vuforia Engine. See the last section of this article for more information about this API. See Vuforia Play Mode for more details on how to replay recordings in Unity.
Capture Session Recordings with Vuforia targets in the scene; not having any visual feedback for target tracking removes any visual confirmation that Vuforia has detected the target.
See the video tutorial to get started quickly with Recording and Playback.
It is also worth mentioning that working out the scenario you wish to capture before recording is essential. A clear strategy on what to capture, whether it is user-centric, sequential steps, or targets under environmental constraints, planning will reward you with an ideal recording that can help in the development process and result in a capable AR application.
Limitations
The following restrictions apply:
- You will need the platform-specific permission to record audio. See Permissions in Session Recorder API Overview.
- Currently, devices with cameras that capture frames in the following pixel formats are supported:
- NV21
- NV12
- YUV420P
- YV12
- Recording and playback of sessions is currently not supported for Ground Plane experiences.
- The SessionRecorder Prefab requires you to add your UI (User Interface) when building applications for Magic Leap.
- Enable Sharing is only supported on iOS and Android devices.
- In addition, make sure you only share with applications that do not modify the recording by compressing the video. Modifying the Session Recording would remove important metadata and make it unusable. OneDrive, Dropbox, and Airdrop all support sharing session recordings without modifying them, while Slack or Google Photos are known to compress videos by default and should be avoided..
- Record only in one view orientation (vertical or horizontal) during a session. Changing the orientation while recording will stop the recording and throw an error. See Record sessions in one view orientation for more details.
Session Recorder in Unity
A readily available Session Recorder GameObject can be added to any application in Unity. It adds a customizable UI for recording and will automatically call the underlying API.
NOTE: For advanced use cases, recording can also be started and stopped using the scripting API. See the section further below for more details.
- First, set up a scene with the Session Recorder GameObject, ARCamera GameObject, and a Vuforia Target. These are easily added by right-clicking in the Hierarchy panel and selecting Vuforia Engine.
Session Recorder configuration
It is possible to edit and add UI and calls to events from the Session Recorder Behavior component. If not needed, you can hide the Clean button or turn off sensor data recording in the Advanced Settings. However, recording sensor data is strongly advised.
The Overlay Canvas contains the UI for mobile devices, while the Holographic Canvas is used on HoloLens. On HoloLens, the voice commands “SessionRecorder Start” “SessionRecorder Stop” and “SessionRecorder Clean” control the Session Recorder’s functionality.
Enable Sharing adds a UI window that allows you to share the recording via cloud storage or other applications on your device.
The Session Recorder Behaviour is fully customizable and can interact with any other MonoBehaviour in the scene. See the Model Target scene from the Vuforia Core Samples for Unity for inspiration.
NOTE: For iOS builds, the UIFileSharingEnabled
plist entry is automatically enabled when the Session Recorder GameObject is added to your scene. You can also configure it manually in Vuforia Configuration.
- Set the build platform and Player Settings according to your connected device and press build and run.
- When the app is launched, it may be required to allow permission to use and access the camera.
- Press the record button once the camera feed is present, and record your scene and Vuforia Target:
Please note the following limitations:- Recording a session is only possible from a device, not from the Unity Editor with Vuforia’s Webcam Play Mode. (However, a Session Recording can be played back in Play Mode, which is described below in Using Recordings)
- Recording a session is currently only possible in landscape left mode.
- Pressing the Stop button will conclude the recording.
- Between recordings, you can remove existing recorded sessions by pressing the Clean button to free up space on the device.
NOTE: The Model Target scene in the Vuforia Engine Core Unity Samples includes an example of a customized recording API.
Transferring Session Recordings
The Session Recording will be stored in the app’s private storage of your device, and there are different procedures to retrieve it depending on the device platform.
It is easier to retrieve a Session Recording on iOS and Android with the Enable Sharing property enabled.
On Android devices
- Use the Enable Sharing in Unity to include a UI for sharing through OneDrive, Dropbox, or any other application that does not modify the recording.
- The Session Recording can be accessed using the
adb
command. The Session Recordings are found within the shared files directory of the app on your device (this is what thegetExternalFilesDir
Android API method returns). The exact path may vary from device to device. One known variant is /storage/emulated/0/Android/data, so the whole path to the Session Recordings might look like
/storage/emulated/0/Android/data/[PACKAGE NAME]/files/Recordings
The section Querying the Location of a Recording in the Session Recorder API Overview article explains how to obtain the path to each Session Recording. The [PACKAGE NAME] will have a form like com.mycompany.MyApps.MyCoolApp (specified within the AndroidManifest.xml of the app using the package
property).
To copy a Session Recording to the current working directory on your computer, you can use the following adb
command :
1Copyadb pull /storage/emulated/0/Android/data/[PACKAGE NAME]/files/Recordings/VuforiaSession-[DATE]-[TIME].mp4 ./
Alternatively, the Session Recording can be copied using Android Studio on Mac or File Explorer on Windows.
In Android Studio, select View -> Tool Windows -> Device File Explorer, browse the media files, right-click them, and select Save As… If no files appear in the Data folder, it may be necessary to right-click and choose Synchronize.
You can use File Explorer to browse the MTP device from This PC for Windows. Connect your device with a USB and select Allow on the pop-up message on the device as it connects. The Session Recordings can now be transferred.
On iOS devices
- Use the Enable Sharing in Unity to include a UI for sharing through OneDrive, Dropbox, Airdrop, or any other application that does not modify the recording.
- The Session Recordings can be accessed using iTunes or macOS Finder. Clicking on the app’s name reveals a number of directories on the main window belonging to the app. The Session Recordings are found within the Documents/Recordings sub-directory that can be saved from iTunes. They can also be dragged onto the Desktop or on MacOS into a directory that is open in Finder.
On UWP devices
- The Session Recordings can be accessed using the operating system’s file browser. The media files are found in the recording app’s private storage space. The path to the Session Recordings directory has the form:
C:\Users\[USER NAME]\AppData\Local\Packages\[PACKAGE NAME]_[PACKAGE FAMILY]\LocalState\Recordings.
The [PACKAGE NAME] will have a form like My.Apps.MyCoolApp and [PACKAGE FAMILY] will be a cryptic string containing numbers and letters.
On the HoloLens
- Use Microsoft’s Device Portal API (see Microsoft's Device Portal API reference article) to download the Session Recording onto a computer. This can be done in two steps using
HTTP GET
calls:- Get a list of all files in the
Recordings
directory, - Get all the files in the list the first call returns.
Please see below for the details of the
HTTP
calls.For step 1:
- URL: http://[HOLO LENS IP]:10080/api/filesystem/apps/files
- Parameters:
knownfolderid = LocalAppData
packagefullname = [PACKAGE NAME]_[PACKAGE FAMILY]
path = \\LocalState\Recordings
(note the double backslashes, which might need to be escaped in a string)
For step 2:
- URL: http://[HOLO LENS IP]:10080/api/filesystem/apps/file (note the singular file, in contrast to files above)
- Parameters:
knownfolderid = LocalAppData
packagefullname = [PACKAGE NAME]_[PACKAGE FAMILY]
path = [PATH TO FILE FROM RETRIEVED LIST]
filename = [FILENAME FROM RETRIEVED LIST]
Please note that you will need to include your HoloLens credentials (username and password) in this
HTTP
call. For the [PACKAGE NAME] and [PACKAGE FAMILY], please see the explanation in the UWP section above.
- Get a list of all files in the
On the Magic Leap
- You can transfer the Session Recordings using the Device Bridge tool from the Magic Leap Lab. For more information, see Magic Leap’s Working with the File System.
- Connect your device and use the Device Bridge to transfer the media files to your desktop.
- The Session Recordings are in the default directory [APP_NAME FOLDER]/files/Recordings.
- All Session Recordings are saved to the [user]/MagicLeap/downloads folder by default.
Using Session Recordings
Using Session Recordings in Unity Editor Play Mode
Using a Session Recording in Unity Editor Play Mode allows you to rapidly prototype your AR experience without building to a mobile device or being close to the physical target you use for testing.
Follow the steps below to use a recording in Play Mode:
- Once the Session Recording is transferred to your workstation, the Vuforia Configuration can be set to Play Mode type – RECORDING.
- Browse to the directory of the saved media files and select the Session Recording you wish to use.
- The Session Recording will be played back once Play Mode is started in the editor.
See Vuforia Play Mode in Unity for more information.
Using Recordings in a mobile application
In cases where you want to test your application on the actual mobile target device but still do not want to test on the physical object/location (e.g., because it is not available), you can play back a recording using the File Driver sample in a native or Unity application.
See the File Driver Sample for more information.
Session Recordings Backward Compatibility
Session Recordings created with Vuforia Engine 10.20 and later are not backward compatible with earlier versions of the Vuforia Engine SDK.
To capture sequences and use them in projects using Vuforia Engine 10.19 or earlier, download the converter tool for your operating system and convert the Session Recordings. Session Recordings are always forward-compatible.
Run the executable with the path to the Session Recording followed by the value 1
.
1Copy./MP4VersionConverter <My_Recording> 1
Creating Recordings using the SessionRecorder API
Using the Unity scripting API
Instead of using the Session Recorder GameObject, the recording functionality can also be used by directly calling the SessionRecorder scripting API. This allows you to create custom interfaces and instructions on what to scan and how long.
Start by calling the SessionRecorder.Instance
and initiate a recording using the SessionRecorder.StartRecording
call. While recording, you can check the status with GetRecordingStatus
and enlist conditions depending on the returned status. Conclude the recording with SessionRecorder.StopRecording
and call the method GeRecordingPath
to identify the location of the stored Session Recording.
For more information, please refer to the Session Recorder API Overview.
Creating Recordings in a native application
To use the Session Recorder in C, start by retrieving the VuSessionRecorderController
using vuEngineGetSessionRecorderController
. You can then create a VuRecording
with vuSessionRecorderControllerCreateRecording
. The recording can either be automatically started on creation or explicitly started with vuRecordingStart
.
To stop the recording, use the vuRecordingStop
call. This will output the recording data as an MP4 file containing the recorded video and sensor data. The file will be named VuforiaSession, followed by date and time values.
The frame images are associated with device pose data and similar information that helps you test and troubleshoot your device and scenario. The types of data that can be recorded are identified with vuSessionRecorderControllerGetSupportedRecordingDataFlags
.
Use vuSessionRecorderControllerGetDefaultRecordingDataFlags
to get the recommended sources for recording on the current device. Using the recommended set of sources will always result in a recording that can be used for AR sequence playback.
After a recording is started, the location of the MP4 file can be queried with vuRecordingGetPath
. To retrieve the file, please see the above section and note the different approaches for different operating systems.
NOTE: If you plan to record with an iOS device, the UIFileSharingEnabled
plist entry must be set in Xcode.
For additional Session Recorder methods, please see Session Recorder API Overview, which illustrates in detail how these can be used with examples.