I'm trying to get the Vuforia Unity plugin working with Stereolabs' ZED stereo camera and its Unity plugin. Vuforia can use it as a direct webcam input and it will track images, however it's displaying feed from the cam as stereo side-by-side (so the user sees both left and right video feeds together).
The ZED SDK, which the ZED Unity plugin interfaces with, not only splits that input into separate textures, but also applies rectification to the image and calculates a new projection matrix from this. Normally, I would use the SDK to display to the user, and Vuforia to track where objects are, and do some fancy matrix multiplication to bring the object positions into the correct position relative to the user's view from the ZED SDK. However, the biggest issue is that you can't have two streams going to the ZED at once.
So my goal is to provide a video feed to Vuforia after the ZED SDK has processed the images. I've seen there's not an automatic way to do that, but it appears that when using a webcam, the Vuforia Unity plugin creates a WebcamTexture which is then sent to Vuforia to use as the base video feed. This is visible from this call stack:
I've been able to get to that texture via Resources.FindObjectsOfTypeAll<WebCamTexture>(), and saw that if I paused that image's updates (without pausing the actual webcam) then it would indeed affect Vuforia's image recognition, which would freeze detected objects according to the last frame. That means that if I could specify my own texture instead of letting Vuforia produce it, I could do exactly what I want.
But that's where I've hit a wall. All the related classes that handle this WebcamTexture are protected, and WebcamTextures themselves seem to be the least malleable form of Texture within Unity. Even being able to get the pointer to that texture doesn't let me do much.
So my question is, how can I either:
1. Modify this texture - I've tried marshaling my own byte array into it, but I crash every time, and can't get much info about the texture's format/size/etc. to find out why
2. Specify my own texture - somehow tell Vuforia to use a different texture (ideally a RenderTexture) instead of the WebcamTexture you create