Log in or register to post comments

Laggy video stream on Unity iOS after set pixel format

September 4, 2018 - 7:42am #1

Hi,

We are trying to solve a video lag problem for a long time without any success. In our app, we are using 

CameraDevice.Instance.GetCameraImage(PIXEL_FORMAT);

inside of 

private void OnTrackablesUpdated()

to capture the camera's image for further processing. To make it work, we need to set the pixel format. On mobile devices (not in the editor) the correct format is

Image.PIXEL_FORMAT.RGB888

Our implementation is very similar to here: https://library.vuforia.com/articles/Solution/Working-with-the-Camera#How-To-Obtain-HD-Camera-Frames.

Since we upgraded from old versions of Unity and Vuforia (Unity 5.6) we have the problem that the camera video stream becomes laggy as soon as the pixel format RGB888 is set. We tried using different pixel formats and unregistering the pixel format after we captured the image but the laggy video remains until vuforia or the scene is reset. This happens on iPhone 7 Plus but not on iPhone X, see versions below. We have only regular 2D Vuforia image targets in our scene. This problem exists only on iOS, on Android and in the Editor the video stream is fine.

Any idea what could cause this problem? Could it be related to ARkit? 

In the video I uploaded here, I move the phone up and down and you can see that after 4 seconds the video starts to lag: https://www.dropbox.com/s/38cn2lzrq1z1l88/IMG_2897.TRIM.MOV?dl=0

 

Versions:

Unity 2018.2.5f1

Vuforia 7.2.23

iPhone 7 Plus, iOS 11.4.1

iPhone X, iOS 11.4.1

Xcode 9.4.1

Laggy video stream on Unity iOS after set pixel format

December 5, 2018 - 8:46am #9

Hi there!

We're having the same problem here. After setting the pixel format the live image gets delayed by about 1-1.5 secs...

This is reproducable on an IPad Pro A1673, with IOS 12.0 - but not on an IPhone 6, running 11.2.

So far I understand, that ARKit changed its version with IOS 12.3, so I rather suspect some relation to this.

Laggy video stream on Unity iOS after set pixel format

September 7, 2018 - 2:18am #8

It's quite instant, you notice it right away. The video stream is overall laggy with some big lag spikes every 5 secs or so.  On this model A1673 / MLMV2FD/A (should be a quite recent new ipad pro) it happens the most.

By experimenting on the project I've sent you I've closed down on the problem:

Deploying on graphics api:

GL2 -> video lags behind movement with big lagspikes every 5 secs or so

Metal -> noticeably laggy but without the spikes

GL3 -> initially i thought it did not lag, after a while lags as much as gl2

 

Anyway I've tried different older ipads and it works fine, FabioZ tried more devices, for me it really lagged only on the above mentioned ipad pro retina.

Laggy video stream on Unity iOS after set pixel format

September 6, 2018 - 2:24pm #7

DisneyResearchZurich wrote:

Here's a couple of tests I've done:

  • ios 11.0  model A1673 / MLMV2FD/A (ipad pro)    -> Video Lags
  • ios 11.4  model A1673 / MLMV2FD/A (actually the same ipad pro)  -> Video Lags
  • ios 11.4  model A1566 -> No lags
  • ios 9.3.5 model ME987GP\A (older ipad)  -> No lags

I'm using:

Unity2017.4.7f1 with the included vuforia version (both metal and gl2)

xcode 9.4.1

I attempted to reproduce this with the project you sent me on an iPad pro with iOS 11.3 and was not seeing the issue. Does the issue happen immediately, or does it take some time before it is noticeable?

Thanks,

Vuforia Engine Support

Laggy video stream on Unity iOS after set pixel format

September 6, 2018 - 8:04am #6

Here's a couple of tests I've done:

  • ios 11.0  model A1673 / MLMV2FD/A (ipad pro)    -> Video Lags
  • ios 11.4  model A1673 / MLMV2FD/A (actually the same ipad pro)  -> Video Lags
  • ios 11.4  model A1566 -> No lags
  • ios 9.3.5 model ME987GP\A (older ipad)  -> No lags

I'm using:

Unity2017.4.7f1 with the included vuforia version (both metal and gl2)

xcode 9.4.1

Laggy video stream on Unity iOS after set pixel format

September 6, 2018 - 7:18am #5

I have the same lag also using metal

Laggy video stream on Unity iOS after set pixel format

September 6, 2018 - 7:09am #4

Hi,

I think the error is because in our project we are using OpenGLES2.0 (for other reasons). I can reproduce the problem with the Vuforia sample project:

- Create new unity project

- Import Vuforia Core Samples from unity asset store, (override all project settings)

- Create new scene with just ARCamera and one ImageTarget (Astronaut)

- Add a Capsule to the ImageTarget

- Add CameraImageAccess script to ARCamera (copied from here https://library.vuforia.com/articles/Solution/Working-with-the-Camera)

- Run in Editor -> works

- Switch platform to iOS

- Switch Graphics API to only OpenGLES2

- Deploy on device. After a few seconds the video stream becomes laggy.

The problem occurs on

- iPhone 7, iOS 11.3.1

- iPhone 7 Plus, iOS 11.4.1

- iPad MP2G2TY/A (not pro), iOS 11.3.1

But it does not occur on

- iPad ML0R2TY/A (pro), iOS 11.3



Versions:

Unity 2018.2.5f1

Vuforia 7.2.23

Xcode 9.4.1

Laggy video stream on Unity iOS after set pixel format

September 5, 2018 - 9:48am #3

Hi,

Thanks for your answer. We were seeing that problem with several iPhones and iPads on iOS11 but not on iOS10, and not with iPhone X (maybe because it has a faster CPU?).

That's a good idea, let me try to recreat the problem with one of your sample projects. I'll let you know as soon as I have a result. 

 

Laggy video stream on Unity iOS after set pixel format

September 4, 2018 - 5:06pm #2

Hello fabioz,

I'm unable to reproduce this issue. Are you only seeing it on the iPhone 7 Plus with iOS 11.4? Do you see it with any other versions of iOS? Could you share the full code you're running in your update function (can direct message it to me if preferred)? Are you able to reproduce this by dropping your code into our samples?

Thanks,

Vuforia Engine Support

Log in or register to post comments