Log in or register to post comments

Getting current image from frame from JNI

December 16, 2012 - 10:59pm #1

hi,

i followed one of post here to get current frame image(for RGB565) from JNI to java as byte array as below:

 

if (image->getFormat() == QCAR::RGB565)
{
 
// for rbg565, the pixels are stored as an array of shorts<br />
 
 
const short* pixels = (const short*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
int numPixels = width * height;
jbyteArray pixelArray = env->NewByteArray(numPixels * 2);
env->SetByteArrayRegion(pixelArray, 0, numPixels * 2, (const jbyte*) pixels);
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
}
 
this works fine for some devices such as samsung galaxy series but same code did not work on htc and sony erricsson devices ,so going furhter i found that these devices give diffrent image PIXEL_FORMAT (YUV and others), so could you please help me out to get image byte array for other PIXEL_FORMAT such as:
 
RGB565
GRAYSCALE
YUV
RGBA8888
UNKNOWN_FORMAT

 

Hi again, I've tested on a

December 21, 2012 - 8:45am #14

Hi again,

I've tested on a few devices here (including Samsung and HTC); but I was unable to get the crash that you reported (using the code snippets as I reported them in the latest messages);

if I can make some suggestions:

- make sure to get a "clean" image target sample and only put the necessary code snippets as I described;

- if you get the crash again, put LOGs at various places to see where exactly it crashes, and what function calls are the "suspicious" ones;

- try to comment out the JNI function call that calls the setRGB565CameraImage on the Java side from ImageTargets.cpp, and see if that is the cause of the problem; more in general, try to remove (or comment out) the lines of code incrementally until you find what is the code lines that are most likely responsible for the crash

Let me know how it goes.

 

Hi yogendra, I mad a few

December 21, 2012 - 7:59am #13

Hi yogendra, 

I mad a few more tests and the reason why the setFrameFormat() has no effect is because the call to 

QCAR::CameraDevice::getInstance().selectVideoMode( QCAR::CameraDevice::MODE_DEFAULT) )  which is in the "_startCamera()" function actually resets that setting;

so, if you move the "setFrameFormat()" to the _startCamera() function right after the |selectVideoMode() function call and for safety also after the CameraDevice::getInstance().start(), then the frame format will be actually set as desired;

so your code in startCamera() should look like in the following snippet:

 

    // ....
 
    // Select the default mode:
    if (!QCAR::CameraDevice::getInstance().selectVideoMode(
                                QCAR::CameraDevice::MODE_DEFAULT))
        return;
 
    QCAR::setFrameFormat(QCAR::RGB888, true);
 
    // Start the camera:
    if (!QCAR::CameraDevice::getInstance().start())
        return;
 
  QCAR::setFrameFormat(QCAR::RGB888, true);

    //  etc ...

 

I verified this on a Samsung by setting various modes (RGB888, RGB565, YUV, etc.);

I'm now going to make a test on an HTC to see if I can get your same crash.

 

hi ,sorry to say that doing

December 21, 2012 - 2:00am #12

hi ,sorry to say that doing again as you said now,htc one x is giving same error i reported ,and one thing more that if i set pixel format in initApplicationNative it has no effect in htc and sony erricsson devices.

i am very confused why my code which perfectly work for RGB565 in samsung giving me such triouble in other devices. 

Hi, I did not observe that

December 19, 2012 - 6:03am #11

Hi, I did not observe that crash in my tests, however that could very well due to a threading issue

(perhaps a conflictual access to the imageRGB565 between the QCAR_onUpdate and the renderFrame);

to fix that you need to put all the relevant image access code in the QCAR_onUpdate, i.e. adjust the code as follows:

1 - remove the global variable QCAR::Image imageRGB565 (we don't need it any more)

2 - add these two global variables (javaVM and activityObj) and the function JNI_OnLoad:

    //global variables

    JavaVM* javaVM = 0;

    jobject activityObj = 0;

 

    JNIEXPORT jint JNICALL

    JNI_OnLoad(JavaVM* vm,  void* reserved)  {

        LOG("JNI_OnLoad");

        javaVM = vm;

        return JNI_VERSION_1_4;

    }

 

3 - Add these two lines of code at the end of the initApplicationNative:

    activityObj = env->NewGlobalRef(obj);

    QCAR::setFrameFormat(QCAR::RGB565, true);

 

 

4 - Remove the code that we previously added in _renderFrame() and move it to the QCAR_onUpdate(); so the QCAR_onUpdate in the end will contain this code:

        QCAR::Image *imageRGB565 = NULL;

        QCAR::Frame frame = state.getFrame();

        for (int i = 0; i < frame.getNumImages(); ++i) {

            const QCAR::Image *image = frame.getImage(i);

            if (image->getFormat() == QCAR::RGB565) {

                imageRGB565 = (QCAR::Image*)image;

                break;

            }

        }

        JNIEnv* env = 0;

        if ((javaVM != 0) && (activityObj != 0) && (javaVM->GetEnv((void**)&env, JNI_VERSION_1_4) == JNI_OK)) {

            if (imageRGB565) {

               const short* pixels = (const short*) imageRGB565->getPixels();

               int width = imageRGB565->getWidth();

               int height = imageRGB565->getHeight();

               int numPixels = width * height;

               jbyteArray pixelArray = env->NewByteArray(numPixels * 2);

               env->SetByteArrayRegion(pixelArray, 0, numPixels * 2, (const jbyte*) pixels);

               jclass javaClass = env->GetObjectClass(activityObj);

               jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");

              env->CallObjectMethod(activityObj, method, pixelArray, width, height);

              env->DeleteLocalRef(pixelArray);

            }

        }

 

Last thing:

in Java, move the method setRGB565CameraImage from ImageTargetsRenderer.java to ImageTargets.java

 

 

 Hi i tried exactly what you

December 19, 2012 - 4:01am #10

 

Hi i tried exactly what you said to do in CloudReco_UpdateCallback and renderframe,

and tried to setFrameFormat(QCAR::RGB565,true) in initApplicationNative and also in renderframe but getting following error in htc one x.

12-19 17:02:40.680: A/libc(12968): Fatal signal 11 (SIGSEGV) at 0x4bce8d00 (code=2)

i think when activity goes in pause or resume somthing happens,sorry i can't figure out exact error as it comes from JNI.

 

meanwhile i am trying cloudreco sample.

 

Here is the code that I just

December 19, 2012 - 2:14am #9

Here is the code that I just tested (assumption is that you have called setFrameFormat( QCAR::RGB565, true ) ):

//Global variable (ImageTargets.cpp)

QCAR::Image *imageRGB565 = NULL;

 

 
In QCAR_onUpdate() (ImageTargets.cpp):
 
//look for the image with format RGB565
QCAR::Frame frame = state.getFrame();
    for (int i = 0; i < frame.getNumImages(); ++i)
    {
    const QCAR::Image *image = frame.getImage(i);
    if (image->getFormat() == QCAR::RGB565)
    {
    imageRGB565 = (QCAR::Image*)image;
    break;
    }
    }
 
 
In _renderFrame() function (ImageTargets.cpp):
 
//Invoke Java method via JNI:
if (imageRGB565)
    {
    char msg[50];
    sprintf(msg, "Image RGB565 found: size: %d x %d", imageRGB565->getWidth(), imageRGB565->getHeight());
    LOG(msg);
 
    const short* pixels = (const short*) imageRGB565->getPixels();
    int width = imageRGB565->getWidth();
    int height = imageRGB565->getHeight();
    int numPixels = width * height;
    jbyteArray pixelArray = env->NewByteArray(numPixels * 2);
    env->SetByteArrayRegion(pixelArray, 0, numPixels * 2, (const jbyte*) pixels);
    jclass javaClass = env->GetObjectClass(obj);
    jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");
    env->CallObjectMethod(obj, method, pixelArray, width, height);
    env->DeleteLocalRef(pixelArray);
    }
 
 
And this is the Java code (ImageTargetRenderer.java):
 
public void setRGB565CameraImage(byte[] buffer, int width, int height) 
    {
    Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.RGB_565);
    bitmap.copyPixelsFromBuffer(ByteBuffer.wrap(buffer));  
    }
 
Let me know if anything is not clear.

 

Hi, the setFrameFormat

December 19, 2012 - 2:08am #8

Hi, the setFrameFormat function actually adds an extra image (the method name is also a bit misleading) to the array of images,

but that is not necessarily located at index 0 (my imprecision here), in practice it can be at index 1 or 2, depending on the device;

so, you will need to search for it with a loop, going to post you the full code (next message...)

 

Hi thanks for your support.i

December 18, 2012 - 11:49pm #7

Hi thanks for your support.

i tried as you suggested but after setting:

 QCAR::setFrameFormat( QCAR::RGB565,  true );

pixel format changed at frame.getImage ( 2 ) inspite of frame.getImage ( 0 ) as you said,i tried it on three different devices.

have you tried to get image bytes from frame on devices?as i am bit confused about it,and worried as how it will behave on other devices. could you pplease suggest me the best way to get image byte array per frame so that i can get bitmap from it on every devices.

i tried as below in renderframe:

 QCAR::setFrameFormat(QCAR::RGB565,true );

QCAR::Frame frame = state.getFrame();
 
for (int i = 0; i < frame.getNumImages(); i++)
{
 
const QCAR::Image* image = frame.getImage(i);      
 
           
(    for frame.getImage(0) no change but changed to RGB565 for frame.getImage(2)    )
 
 
LOG("RGB565_pixal===========%d",image->getFormat());
if (image->getFormat() == QCAR::RGB565)
{
LOG("RGB565_pixal===========");
// for rbg565, the pixels are stored as an array of shorts<br />
 
const short* pixels = (const short*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
int numPixels = width * height;
jbyteArray pixelArray = env->NewByteArray(numPixels * 2);
env->SetByteArrayRegion(pixelArray, 0, numPixels * 2, (const jbyte*) pixels);
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
}

Hi, if you don't want to deal

December 18, 2012 - 6:45am #6

Hi, if you don't want to deal with different frame formats and just want to focus on one, you could enforce the use of a given format, using:

  QCAR::setFrameFormat( desired_pixel_format,  true );

for example, to enforce the use of RGB565 (on all devices), you could use:

   QCAR::setFrameFormat( QCAR::RGB565,  true );

 Then the image with that format should be stored as the first Image in the Frame object (i.e. at index zero), so it can be retrieved using:

    const QCAR::Image *image = frame.getImage ( 0 );

I hope this helps. Let me know if you still have problems after using that.

 

is there is any one format

December 18, 2012 - 4:02am #5

is there is any one format which i can use for diffrent devices by finding in loop,since different devices shows different behavier i am suspicious about those on which i had not tested.

what is the best way to capture single image byte data so that i can convert it to bitmap at every frame on all devices?

thanks for your support........

Hi, as you said, each frame

December 18, 2012 - 1:33am #4

Hi,

as you said, each frame contains multiple images (frame.getImage(0), frame.getImage(1), and so on), and each of these images is at a different format (and also different resolution, by the way; for example you might get the first 2 images at 640 x 480 in RGB565 and Grayscale, then for example the third image in grayscale at 320 x 240, etc...);

I'm not quite sure what is your goal with retrieving those images; do you need all of the images in each frame? or can't you just take the first one (i.e. the one at index zero, using  frame.getImage( 0 ))  ?

Then you should be able to retrieve your image and handle it with your code;

Some notes about your code snippet:

1 - I see that you always use  "const short* pixels = (const short*) image->getPixels();" to store your buffer, but that is not ideal for RGB888 for instance (as "short" is two-bytes long, while a pixel is 3 bytes in the RGB888 case; this could lead sometimes to have your last array element to point to a memory block which is invalid in its second half byte; that could theoretically be a source of problems);    I would generally recommend to use < const char* > instead of < const short * > (considering that you are going to copy the buffer anyway using setByteArrayRegion) 

2 - in all your different cases, I see you always call the function "setRGB565CameraImage" (except in thr YUV case); this is not correct in the case of RGB888 or others

3 - I would use " else if " instead of " if " to handle the different cases

I hope this helps.

 

 

Hi AlessandroB thanks for

December 17, 2012 - 9:20pm #3

Hi AlessandroB thanks for your reply.

now i am able to get byte array in java for YUV also but my exact prblem is that, different devices give different pixel formats for every image in frame. samsung give RGB565 and GRAYSCALE for images in frame but htc and Sony Ericsson give YUV ,and GRAYSCALE for different images in frame so on...... , so for one loop different condion satisfy and for another loop diffrent . hope you understand my problem.

i used following code to get all type of pixel formats and successfully converted each to bitmap in java but code crashed after using them together:

if (image->getFormat() == QCAR::RGB565)

{
//LOG("RGB565_pixal===========");
// for rbg565, the pixels are stored as an array of shorts<br />
 
const short* pixels = (const short*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
int numPixels = width * height;
jbyteArray pixelArray = env->NewByteArray(numPixels * 2);
env->SetByteArrayRegion(pixelArray, 0, numPixels * 2, (const jbyte*) pixels);
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
}
if (image->getFormat() == QCAR::YUV)
{
 
//LOG("YUV_pixal===========");
 
const short* pixels = (const short*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
//LOG("yuv %d%d", width, height);
int numPixels = width * height;
 
jbyteArray pixelArray = env->NewByteArray(numPixels * 1.5);
env->SetByteArrayRegion(pixelArray, 0, numPixels * 1.5, (const jbyte*) pixels);
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setYUVCameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
 
}
if (image->getFormat() == QCAR::RGB888)
{
LOG("RGB888_pixal===========");
 
 
const short* pixels = (const short*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
int numPixels = width * height;
 
jbyteArray pixelArray = env->NewByteArray(numPixels * 3);
env->SetByteArrayRegion(pixelArray, 0, numPixels * 3, (const jbyte*) pixels);
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
}
if (image->getFormat() == QCAR::RGBA8888)
{
LOG("RGBA8888_pixal===========");
 
const short* pixels = (const short*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
int numPixels = width * height;
 
jbyteArray pixelArray = env->NewByteArray(numPixels * 4);
env->SetByteArrayRegion(pixelArray, 0, numPixels * 4, (const jbyte*) pixels);
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setRGB565CameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
}

there is any robust way to get image byte array from frame which works on all devices?

i am bit stuck at this for few days please help me out.....

Hi, for RGB888 you will need

December 17, 2012 - 7:42am #2

Hi,

for RGB888 you will need to adjust the code as follow:

 

if (image->getFormat() == QCAR::RGB888)
{
const char* pixels = (const char*) image->getPixels();
int width = image->getWidth();
int height = image->getHeight();
int numPixels = width * height;
jbyteArray pixelArray = env->NewByteArray(numPixels * 3); // RGB888 => 3 components per pixel !
env->SetByteArrayRegion(pixelArray, 0, numPixels * 3, (const jbyte*) pixels); // RGB888 => 3 components per pixel !
jclass javaClass = env->GetObjectClass(obj);
jmethodID method = env->GetMethodID(javaClass, "setRGB888CameraImage", "([BII)V");
env->CallObjectMethod(obj, method, pixelArray, width, height);
env->DeleteLocalRef(pixelArray);
}

 

On the Java side, you can use some code similar to the following:

 

bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);

bitmap.copyPixelsFromBuffer(ByteBuffer.wrap(image_bytes));     

For other formats like YUV, you will need to implement some conversion as I don;t think the Bitmap class supports that format natively;

such conversion should not be difficult in principle (there are simple maths functions to convert YUV to RGB), however it may result a bit expensive to do, as you will need to iterate over each pixel.

Let me know if you need some help with the YUV conversion step.

 

Log in or register to post comments