Log in or register to post comments

Changing WebcamTexture

February 19, 2019 - 5:25pm #1

I'm trying to get the Vuforia Unity plugin working with Stereolabs' ZED stereo camera and its Unity plugin. Vuforia can use it as a direct webcam input and it will track images, however it's displaying feed from the cam as stereo side-by-side (so the user sees both left and right video feeds together). 

The ZED SDK, which the ZED Unity plugin interfaces with, not only splits that input into separate textures, but also applies rectification to the image and calculates a new projection matrix from this. Normally, I would use the SDK to display to the user, and Vuforia to track where objects are, and do some fancy matrix multiplication to bring the object positions into the correct position relative to the user's view from the ZED SDK. However, the biggest issue is that you can't have two streams going to the ZED at once. 

So my goal is to provide a video feed to Vuforia after the ZED SDK has processed the images. I've seen there's not an automatic way to do that, but it appears that when using a webcam, the Vuforia Unity plugin creates a WebcamTexture which is then sent to Vuforia to use as the base video feed. This is visible from this call stack: 

 

UnityEngine.WebCamTexture:Play()

Vuforia.WebCamTexAdaptor:Play()

Vuforia.WebCam:StartCamera()

Vuforia.CameraDevice:StartCameraDevice()

Vuforia.CameraDevice:Start()

Vuforia.VuforiaARController:StartVuforia(Boolean)

Vuforia.VuforiaARController:StartAfterInitializationCompleted()

 

I've been able to get to that texture via Resources.FindObjectsOfTypeAll<WebCamTexture>(), and saw that if I paused that image's updates (without pausing the actual webcam) then it would indeed affect Vuforia's image recognition, which would freeze detected objects according to the last frame. That means that if I could specify my own texture instead of letting Vuforia produce it, I could do exactly what I want. 

But that's where I've hit a wall. All the related classes that handle this WebcamTexture are protected, and WebcamTextures themselves seem to be the least malleable form of Texture within Unity. Even being able to get the pointer to that texture doesn't let me do much. 

So my question is, how can I either: 

1. Modify this texture - I've tried marshaling my own byte array into it, but I crash every time, and can't get much info about the texture's format/size/etc. to find out why

or

2. Specify my own texture - somehow tell Vuforia to use a different texture (ideally a RenderTexture) instead of the WebcamTexture you create



Thanks! 

Changing WebcamTexture

April 16, 2019 - 8:01am #3

Hi Medabit, thanks for the explanation. I'll take a look at the External Camera option. 

Changing WebcamTexture

March 14, 2019 - 3:33pm #2

Hello,

Unfortunately, I think you're trying to overcome product issues that the Vuforia Engine SDK is not meant to solve. The Unity+Vuforia+webcam+PC use case is specific to Unity's PlayMode, and only as a convenience feature to developers. If I understand your description correctly, the SDK is not meant to be used in that fashion as thus the "roadblocks" you're finding are deliberate product decisions.

That being said, we recently opened up the SDK through the External Camera feature. This allows camera frames to be fed into the SDK from external sources, and thus may meet your requirements. The only caveat is that the External Camera has been developed for, and tested upon, Android, iOS and UWP devices only.

Suggest you evaluate the feature to see if it meets your needs.

Thanks,

Vuforia Engine Support

Log in or register to post comments