Have you given AsyncGPUReadback a shot yet?  https://docs.unity3d.com/ScriptReference/Rendering.AsyncGPUReadback.Request.html

This example on github is a good place to start: https://github.com/keijiro/AsyncCaptureTest

I honestly don't if these methods are supported on HoloLens or if it'd be any faster, but it may be something to try.



 

I did not know about this function, but it appears to need a Texture or a ComputeBuffer as source, which would be great, if I had either. 

Is there a reason you can't use the video background texture?  Ah, duh, you're on Hololens, I see why now.

Maybe Vuforia still retains the video background texture in memory, maybe make sure of that before totally discounting this.  

 

I figured that it is less about getting the data but actually showing it. I can get the raw data with CameraDevice.Instance.GetCameraImage(pixelFormat).pixels without much delay since nothing gets copied. Using Texture2D.loadImageFromRawData() is equally slow (maybe even used by Vuforia internally) to create a texture and since I have to copy 856 * 504 * 3 bytes of data in one frame, because writing to (at least) Texture2D can only happen on the main thread. 

Anyway what I am actually trying to achieve is to send the augmented view of the user to a laptop. I have a dedicated camera in the scene that renders everything to a render texture, which gets encoded to a byte array and send via UDP. Single problem is that I am not able to get the background in a reasonable way. Sending the raw data to the laptop and assemble the scene there is also not an option (would require 19 UDP packages to send one background image ... I don't want to do that via UDP)

And you can't use the built in live streaming hololens features cause Vuforia needs access to the camera right?  

You might be hitting the technical limitations of things at this point.

Hello,

Based upon experiments with HoloLens while running Vuforia, here is what we know about the Mixed Reality Capture (MRC) feature:

1. You can record videos of Vuforia apps running on HoloLens. Note that for this to work, the Vuforia performance mode must be set to MODE_OPTIMIZE_SPEED, which sets the camera capture resolution to be 720p, e.g. 1280x720.
2. The app running Vuforia must be started *before* you started to record with HL’s MRC. If you start recording first, the video will abruptly stop once Vuforia gets access to the HL camera.
3. The MRC capture resolution must be set to the same as the Vuforia performance mode (720p)
4. You *cannot* do MRC Live preview while a Vuforia app is running in any case. If you try to start Live preview while a Vuforia-based app is running on HL, you will get an error in MRC. If you try to start a Vuforia-based app while Live preview is running, the app will not start.

We've had mixed success with these steps in RS4, with results in that it works every time to it works once in a while.

The limitations above are related to real-time camera access for apps running on the HoloLens. In order to guarantee optimal detection and tracking performance, Vuforia is granted sole access to the live camera feed. This is similar to other platforms that we support, such as Android and iOS. You can see that it also works the other way around for apps that do not run Vuforia. The Live preview app requires real-time camera access, so while it is running Vuforia is blocked from accessing the camera.

Thanks,

Vuforia Engine Support

So I implemented a reasonable running live preview, but I have a problem regarding the alignment between the virtual elements and the video feed. Running Vuforia on my laptop I noticed that the VuforiaBackground is using some scaling and translation for proper alignment. On the hololens it seems like I get the virtual Elements seen from my right eye and the camera feed has a bigger field of view. How do I get the transformation matrix to align the background with the virtual objects? I could go with try and error but that seems a bit fiddly .