Topic locked

Unity 4.5 Pro, OnRenderImage Memory Leak

July 8, 2014 - 1:01pm #1
Hello.
 
I develop Android application for recognition target image and render 3d models. I use Unity Pro v.4.5, and Vuforia Unity Extention 3.0.6
 
When I have used ARCamera.prefab from downloaded prefabs, I noticed strange behaviour in my script. When I add this c# script on my ARCamera prefab:

 

public class TestCapture : MonoBehaviour {


	void OnRenderImage(RenderTexture src, RenderTexture dest)
	{
	}
}

and deploy project to my phone (tests on Nexus 5 and Nexus 7) I receive memory leak and about ~350fps. When I remove it, I receive a good ~10-30fps.

I read all post in forum on this subject

https://developer.vuforia.com/forum/unity-3-extension-technical-discussion/unity-image-effects-memory-leak

https://developer.vuforia.com/forum/unity-3-extension-technical-discussion/onrenderimage-qcar-slowdown

But I don't found solution for this issue.

P.S. If use UnityEngine.Camera and attach this script to him it has not memory leak.

Thanks.

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 8:17am #13

Nice!

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 8:08am #12
Thanks Alessandro! You made my day =) 
 
I solve my task using your advice and integrated Everplay! =)

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 7:49am #11

The second camera (Orthographic camera) will only draw the video-background, while the AR Camera will draw the models;

what you need to use the same concept (i.e. rendering the camera texture to a plane), but with a slightly different approach, in which you do not have the Orthographic camera, i.e.:

- have 1 ARCamera, with the Depth parameter set to 0

- disable video backgrounding drawing, as shown in the sample and as explained here too:

https://developer.vuforia.com/forum/faq/unity-how-can-i-disable-camera-video-background-rendering

- add a second perspective Camera (a standard Unity camera) as a child of the AR Camera, with Clear Flags setting set to Solid Color

- the second camera must be oriented exactly like the parent one (the AR Camera) and must be placed at the local origin of the parent 

- add a script to the child perspective camera that ensures to set the filed of view of this camera to match the parent camera (i.e.e the ARCamera)

- render a plane located at some distance in front of the camera, and apply the camera texture to it; note, you need to make the plane fit the camera view, this may require some computation

This will require a little bit of coding, but we tested this approach and it is technically possible to achieve the above (although we don't have a step-by-step tutorial at the moment).

 

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 7:05am #10
Thanks for help. 
 
I have one small question =)
I see BTA example for unity. In this example used two cameras, first ARCamera for recognise target image and second for applied texture. 
I don't clear understand how will work second camera. It will receive augmented models or only texture?

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 6:44am #9

But I can't use Background Texture Access =( Because I don't want applying such texture to a screen -aligned plane.

To clarify this: I am not saying that the ultimate goal of this is to render a screen-alined quad; the approach above (BTA) is meant to render the camera video background in a different way than the "standard" one used in Vuforia; 

this means that you would then be able to use the Clear Flags = Solid Color on your camera,

and then you could still render the 3D augmentation on top of it,

and so, in the end, you could still apply your technique using the OnRenderImage() method to capture the enitre rendered view;

the only difference is that in this case you would have the optimal Clear Flags applied (i.e. = Solid Color), and you should consequently get good results in terms of Frame rate.

 

If the approach above sounds too complex for you,  you could consider using the Render Texture approach described here:

http://docs.unity3d.com/Manual/class-RenderTexture.html

OR:

you may want to consider using existing video capturing tools (I know there exist several for Unity, probably also including free ones).

this is just an example:

https://developers.everyplay.com/documentation/Everyplay-integration-to-Unity3d-game

 

 

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 6:24am #8

Thanks for you detailed response.

But I can't use Background Texture Access =( Because I don't want applying such texture to a screen -aligned plane.

I will try explain what I want =)
My goal is capture video from vuforia app. It will realy great solution if you can capture video with augmented reality and share it with your friends (for example).
I found solution for capture frame on Android.
 
 
And for this solution I need override OnRenderImage method.
 
void OnRenderImage(RenderTexture src, RenderTexture dest)
    {
        if (inProgress && Time.time > nextCapture) {
            CaptureFrame(src.GetNativeTextureID());
            nextCapture += 1.0f / videoFrameRate;
        }
        Graphics.Blit(src, dest);
    }
 

And It realy work! But I get veeery slow video =)

May be you can give the cue another method for get RenderTexture?

 

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 5:49am #7

Just checked with our tech team here, and this is in fact a known issue:

in fact, when using the OnRenderImage(), the Unity engine will execute some extra rendering processing steps which are optimized to run smoothly when the camera Clear Flags are set to "Solid Color" (and probably also working with the "Skybox" setting as well).   Note that the "Solid Color" is usually the default setting for a standard Unity camera (and this is why you see good frame rate when using a normal Unity camera).

however, the Vuforia ARCamera internally sets the camera "Clear Flags" to "Depth Only" (note: this is done at runtime, i.e., even if you set Clear Flags = Solid Color in the ARCamera inspector panel,  at run-time the Clear Flags will be automatically reset to "Depth Only"). This is because  the AR Camera needs to render 3D content on  top of the video-background image (otherwise the video-background image would be cleared away, as you can imagine).

So, due to the use of "Depth only" (as opposed to "Solid Color"), the Unity rendering pipeline may deliver low frame rates (note: the results can vary significantly from one device to another).

Rather than a bug, this is simply a constrain of the current design of the AR Camera.

Workarounds:

It should be possible to achieve good results if you use the rendering approach illustrated in the Background Texture Access sample, i.e. by capturing the camera texture and applying such texture to a screen -aligned plane.  

 

 

 

 

 

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 5:26am #6

Testing right now... I am also getting variable results depending on the device used (e.g. smooth / good FPS on an S3, low FPS on an Asus Memo Pad, for example).

In the meantime, have you tried switching the Camera Mode from MODE_DEFAULT to MODE_OPTIMIZE_SPEED ? (you can do that in the Inspector of the ARCamera).

 

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 4:27am #5

I use Unity Profiler for check memory.

What about app, when I running it, I receive a very large delay for frame ( about one frame in 1,5 seconds )

See profile screenshots: 1st it without OnRenderImage method. Secont with OnRenderImage.

It is worth noting that this behavior is observed only on Android (I did not test on iOS). 
If you run through all of Unity Editor all ok.

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 3:35am #4

I'm just trying to do the same (also with and without the call Graphics.Blit()), and tracking the memory usage with using ADB on the console (adb shell dumpsys meminfo),

but the memory usage I get seems stable, i.e. it gorws a bit in the first few seconds, and then it remain stable (testing with an Image Target sample in Unity).

 

How do you measure the leak ? 

do you end up with a crash after running it for some time ?

 

Concerning the frame rate, you mention 350 fps.... do you mean maybe 3.5 fps ?

 

 

 

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 2:55am #3

Hi, Alessandro.

Yes, I tried call

Graphics.Blit(src, dest);

Inside OnRenderImage, but I receive memory leak too. And If I call empty method, I whatever receive a memory leak.

Have any ideas? 

Thanks!

Unity 4.5 Pro, OnRenderImage Memory Leak

July 10, 2014 - 12:19am #2

Hi,

are you executing any specific code inside the OnRenderImage() method ?   or is it just empty as you posted it ?

Asking because usually you are supposed to call it this way:

void OnRenderImage(RenderTexture src, RenderTexture dest)
{
    Graphics.Blit(src, dest);
}

 

 

 

Topic locked