Log in or register to post comments

Converting YUV image to RGB

July 26, 2016 - 3:52pm #10

Hello,

I am using CameraImageAccess to set the frame format to YUV (required format for other purposes), then later on grabbing frames. In some instances, I need to convert the frame to RGB to copy into a Texture2D. It is important that the frame format remain YUV (as opposed to changing frame format to RGB888) for performance reasons and so I grab the correct frame when it is requested (changing format takes a few frames). So, I have a method to convert the byte array of the YUV frame to RGB. 

What I am ending up with on conversion is a black and white image. It appears I am extracting the Y values correctly based on that, but I continue to see 0 for all my U and V values. Are the YUV formatted frames expected to be black and white, or could I be extracting the U and V values incorrectly? Here is the code I am using to pull these values (based on a post here: http://stackoverflow.com/questions/12469730/confusion-on-yuv-nv21-conversion-to-rgb):

int frameSize = width * height;

int ii = 0;
int ij = 0;
int di = +1;
int dj = +1;

int a = 0;
for (int i = 0, ci = ii; i < height; ++i, ci += di) {
   for (int j = 0, cj = ij; j < width; ++j, cj += dj) {

        int y = (((int) yuv[(ci * width + cj)]));
        int v = (((int) yuv[frameSize + (ci >> 1) * width + (cj & ~1) + 0]));
        int u = (((int) yuv[frameSize + (ci >> 1) * width + (cj & ~1) + 1]));
        y = y < 16 ? 16 : y;

 

Below that code, I calculate the RGB values, but I have confirmed the U and V are always 0, so there is no color. 

Thanks for your help.

Converting YUV image to RGB

July 27, 2016 - 6:28am #9

These results are on the Android platform.

Converting YUV image to RGB

July 27, 2016 - 12:38pm #8

I have also printed out the entirety of the YUV bytes array, and everything after the frameSize (width * height), where the U and V values should be located are all 0's. 

Converting YUV image to RGB

August 3, 2016 - 9:38am #7

Hi MerlinMo,

You should be able to use CameraDevice::SetFrameFormat on multiple formats, this will cause the SDK to do the color conversion for you.  I'm not sure why you are seeing zeros in the UV planes, that may be an unrelated bug.  Does this camera preview come through in proper color?

//mwynkoop

Converting YUV image to RGB

August 3, 2016 - 11:07am #6

We make this setting by calling CameraDevice.Instance.SetFrameFormat(Image.PIXEL_FORMAT.YUV, true).

On both iOS and Android the Vuforia video background texture has color when format is YUV.

Converting YUV image to RGB

August 3, 2016 - 11:19am #5

In addition, you can also add

 

CameraDevice.Instance.SetFrameFormat(Image.PIXEL_FORMAT.RGB565, true);

 

And both color formats should be available. 

 

//mwynkoop

Converting YUV image to RGB

August 4, 2016 - 7:18am #4

Thanks for the suggestion, however the issue with changing formats is that it takes several frames to apply, so the image we capture will be late. We also need the YUV formatted pixel buffer for other purposes, and it is not coming with color as it should (on either Android or iOS).

Converting YUV image to RGB

August 4, 2016 - 7:29am #3

We'll look into why the YUV formats are not being transferred to Unity properly, however I think your best bet in the short term is going to be requesting RGB565 color conversion as discussed below for the entirety of your AR session (or at least in the rough period in which you need access to RGB frames) rather than trying to enable/disable it on a frame-by-frame basis.

 

Also, please provide a code snippet you are using to obtain the YUV frames in which you see the problem so that I may attach it to the bug report.  As well as SDK version and any other important information relating to reproduction.

Converting YUV image to RGB

August 12, 2016 - 12:52pm #2

The SDK version is the current version as of the posting, v5.5.9

Problem: When requesting YUV data on iOS or Android device UV chroma data is zero

1) activate YUV pixel format

CameraDevice.Instance.SetFrameFormat(Image.PIXEL_FORMAT.YUV);

2) retrieve camera pixel data

Image image = CameraDevice.Instance..GetCameraImage(Image.PIXEL_FORMAT.YUV);

3) inspect pixel data

for (int i=image.BufferWidth * image.BufferHeight; i < image.Pixels.Length; i++)

   if (Convert.ToInt32(image.Pixels[i]) != 0) Debug.Log("found non-zero UV data");

Converting YUV image to RGB

August 15, 2016 - 7:54am #1

MerlinMo,

We've identified the issue in the Vuforia Unity plugin and will have a fix in the next Vuforia patch

Thanks,

//mwynkoop 

Log in or register to post comments