Topic locked

Video background configuration

January 31, 2014 - 2:01am #1

Hi,

 

I'm trying to set/understand the video capture in Vuforia : 

In the method : 

 // Configures the video mode and sets offsets for the camera's image
    private void configureVideoBackground()
    {
        CameraDevice cameraDevice = CameraDevice.getInstance();
        VideoMode vm = cameraDevice.getVideoMode(CameraDevice.MODE.MODE_DEFAULT);
        
        VideoBackgroundConfig config = new VideoBackgroundConfig();
        config.setEnabled(true);
        config.setSynchronous(true);
        config.setPosition(new Vec2I(0, 0));
        
        int xSize = 0, ySize = 0;
        if (mIsPortrait)
        {
            xSize = (int) (vm.getHeight() * (mScreenHeight / (float) vm
                .getWidth()));
            ySize = mScreenHeight;
            
            if (xSize < mScreenWidth)
            {
                xSize = mScreenWidth;
                ySize = (int) (mScreenWidth * (vm.getWidth() / (float) vm
                    .getHeight()));
            }
        } else
        {
            xSize = mScreenWidth;
            ySize = (int) (vm.getHeight() * (mScreenWidth / (float) vm
                .getWidth()));
            
            if (ySize < mScreenHeight)
            {
                xSize = (int) (mScreenHeight * (vm.getWidth() / (float) vm
                    .getHeight()));
                ySize = mScreenHeight;
            }
        }
        
        config.setSize(new Vec2I(xSize, ySize));
        
        Log.e(LOGTAG, "Configure Video Background : Video (" + vm.getWidth()
            + " , " + vm.getHeight() + "), Screen (" + mScreenWidth + " , "
            + mScreenHeight + "), mSize (" + xSize + " , " + ySize + ")");
        
        Renderer.getInstance().setVideoBackgroundConfig(config);
        
    }

We get the video capture resolution and the screen resolution, and compute the biggest size possible keeping the video capture ratio.

Is this size be used for the capture from the camera ? if not, what is it used for ?

 

Then, VideoBackgroundTextureInfo class get the image size and the texture size, but what is the relation with the video size ?

The image size seems to be the video capture size at the begin, so the config set sooner is lost ?

I saw the BackGroundTextureAccess example, where the VideoBackgroundTextureInfo is used to adapt the U/V ratio and coordinates, but I dont understand why the VideoBackgroundConfig is not used for this computation, and the relation between the congif of the videobackground and the and texture where it is rendered...

Thanks a lot!

 

Video background configuration

July 22, 2014 - 2:28am #25

Hi ALTernative,

you posted same question here:

https://developer.vuforia.com/forum/ios/how-use-hd-camera-view-record-video

could you follow up there ?

 

Video background configuration

July 21, 2014 - 7:27pm #24

Why video background size is always 640x480 (x360) ? Where is HD support ? In all SDK versions not work HD video background :( [tested on ASUS TF300T and iPad4]

Can somebody help to solve this problem ?

Video background configuration

May 14, 2014 - 6:21am #23

Yes, i knew it ;-)

I just add the setter as a workaround, but it is the right way of doing this.

Because I have my own separate engine, I set the viewport in the 3D engine not in the AR's implementation, so the BTA sample is just inspiration and understanding, not something to follow with precision.

 

BTW, thanks a lot, let's finish this integration, hoping no more bugs ^^

Video background configuration

May 14, 2014 - 6:09am #22

You're right !  My previous statement was not fully correct.   The things is that the viewport is changed as a "side effect" of the bidnVideoBackg function call in some cases; however this should be only occurring on some devices (it basically depending on the type of GPU) due to some internal OpenGL processing which can occur to process the camera image (although I cannot provide here the full details of this).

Anyway, for the application developer (you) this means that this is just one more reason to set your GL viewport right after the call to bindVideoBackground call, like in this line of code taken from the BTA sample (note: in the sample this is called after the bindVideoBackground):

GLES20.glViewport(viewportPosition_x, viewportPosition_y,
            viewportSize_x, viewportSize_y);

So, normally, if your viewportPosition_x/y and viewportSize_x/y are correctly calculated in the configureVideoBackground() , the viewport should be correctly set for your device screen and camera...

 

Video background configuration

May 14, 2014 - 5:31am #21

AlessandroB wrote:

The bindVideoBackground() function simply binds an OpenGL texture, so it does not alter the GL viewport per-se; 

So, here is my sample test : 

 m_viewport = ByteBuffer.allocateDirect(4 * 4).order(ByteOrder.nativeOrder()).asIntBuffer();
            Gdx.gl.glGetIntegerv(GL20.GL_VIEWPORT, m_viewport);
            m_viewport.rewind();
            m_viewport.get(buff);
            Gdx.app.error("Viewport size before", buff[2] + "|" + buff[3]);

			// Bind the texture on index
			Renderer.getInstance().bindVideoBackground(renderingEngine.getImageIndex());

            m_viewport = ByteBuffer.allocateDirect(4 * 4).order(ByteOrder.nativeOrder()).asIntBuffer();
            Gdx.gl.glGetIntegerv(GL20.GL_VIEWPORT, m_viewport);
            m_viewport.rewind();
            m_viewport.get(buff);
            Gdx.app.error("Viewport size after", buff[2] + "|" + buff[3]);

And the output : 

Viewport size before﹕ 1794|1080
Viewport size after﹕ 1280|960

 

Are there some other settings I missed ? 

 

Thanks.

 

Video background configuration

May 14, 2014 - 3:20am #20

In Vuforia, using the bindVideoBackground() and of course all the necessary things to enable AR detection and tracking, are there any change applied to the GLViewport ?

The bindVideoBackground() function simply binds an OpenGL texture, so it does not alter the GL viewport per-se; 

however,as the BTA sample code shows, you should then take care of setting the GL viewport in the correct way, using the viewport origin and size parameters that are computed in the configureVideoBackground() function (again, see the sample code).

 

Video background configuration

May 14, 2014 - 3:04am #19

So, I think I got it, the BTA sample is a good starting point but there is still an issue, only on a HD screen device, and because I have no idea of the issue here is the question : 

 

In Vuforia, using the bindVideoBackground() and of course all the necessary things to enable AR detection and tracking, are there any change applied to the GLViewport ?

 

In my case, with the HD device, the first frame is good, then the Viewport size is set to the camera's size... without any reason, without any changes in the rendering engine... 

Thanks!

Video background configuration

May 7, 2014 - 1:49am #18

Ok thanks :-)

Still investigating to understand everything... 

Video background configuration

May 7, 2014 - 1:46am #17

That's just because the BTA sample uses a special Shader which allows performing image distortion by touching the screen and dragging your finger across the screen;

in order to achieve this "distortion" effect, the Shader acts on the vertices of the BG mesh; this is the reason why the mesh is buit that way.

But of course, yes,  if you don't need to achieve such special visual FX, i.e. if you just need to render the camera image "as is", then a "single-quad" mesh would be enough.

 

Video background configuration

May 7, 2014 - 1:38am #16

About the BTA sample : 

 

Why does the BgMesh is divided in cols/rows ? 

A single quad is enough to display a texture. No need to subdivide the plane.

Video background configuration

May 6, 2014 - 8:33am #15

VideoBGTextureInfo is more "informational"; you might use it for special purpose, e.g. if your app needs to know what is the camera image size and/or what is the OpenGL texture size for the camera image.

But typically, VideoBGConfig  is the "important" one, i.e. the  class you should care of (as shown in the configureVideoBackground sample function).

Note that at OpenGL level, as shown in the BTA sample, the VideoBGConfig is used for things like setting the GL viewport, so it's not just about the camera texture, but there are a bunch of OpenGL settings that are affected by this class.

 

 

Video background configuration

May 6, 2014 - 8:13am #14

Yes, got it, but I don't understand what I get binded after all that.

 

The VideoBGConfig set a size that allow a good fullscreen render, and the VideoBGTextureInfo does something not at all related.

If they are not at all related, what are they respectively used for ?

 

Thanks

Video background configuration

May 6, 2014 - 7:38am #13

If we read the API reference, for VideoBackgroundTextureInfo class, we have 2 relevant methods:

com.qualcomm.vuforia.VideoBackgroundTextureInfo.getImageSize( )

Width and height of the video background image in pixels
Describe the size of the image inside the texture. This corresponds
to the size of the image delivered by the camera 

com.qualcomm.vuforia.VideoBackgroundTextureInfo.getTextureSize( )

Width and height of the video background texture in pixels
Describes the size of the texture in the graphics unit
depending on the particular hardware it will be a power of two
value immediately after the image size 

 

This is telling that the size of the VideoBackgroundTextureInfo corresponds to the Camera image size

(and for the underlying OpenGL texture, this texture size may be rounded up to the next power-of-two; for example, if the camera delivers 900 x 500 pixels, the underlying OpenGL texture may have a size of 1024 x 512)

 

On the other hand, the VideoBackgroundConfig is used to specify a size which corresponds to the screen resolution, with some little extra adjustment (scaling) to fit the enitre screen (see the configureVideobackground function in the sample).  This is usually different (and can be significantly different) from the VideoBackgroundTextureInfo data.

 

 

Video background configuration

May 6, 2014 - 7:19am #12

Hi !

New update, still some issues and missunderstanding : 

 

I the documentation : https://developer.vuforia.com/resources/api/classcom_1_1qualcomm_1_1vuforia_1_1_video_background_texture_info#a56e4b1e9b884b3a9b8daef5d254e5a16 

The Image size in the texture is described as the camera size.

But the VideoBackgroundConfig changes it size to allow the camera beeing rendered on the whole screen size... So here is the question : the VideoBackgroundTextureInfo.Image.size is the same than VideoBackgroundConfig.size or DisplayMetrics ?

Maybe the documentation is not very clear about that settings.

 

Thanks :)

Video background configuration

February 21, 2014 - 3:49am #11

Yes I saw that, but I cannot translate my viewport, so I just move the orthographica camera, that should do the same.

I'll try to make a complete processing report to be sure of each action done, and to make it work once for all !

Thanks!

Video background configuration

February 21, 2014 - 3:46am #10

You are almost there.

But if you check the BTA sample code, you will see that we also act on the glViewport.   In particular, we set the viewport origin to some value which may be negative, basically to clip away some of the extra image.

So, the bricks to put together are:

- gl viewport

- texture size

- camera video size

- UV coords

 

Video background configuration

February 21, 2014 - 12:57am #9

Ok, let's sum up, and don't hesitate to make corrections please:

  • First the 
configureVideoBackground()

method configure the VideoBackgroundconfig accrording to the camera and the screen size.

This way, we have a video resolution a lit larger in height or in width than the screen size. This means the image will be cropped later, the get only the interessting part wihtout loosing the camera ratio.

example : 

Video camera (640 , 480), Screen (854 , 480),

So the camera picture size will be set to (854 , 640)

  • Next, this picture will be binded in a power of two sized texture

So a (1024, 512) texture size will support the camera picture, sizing (854 , 640). Let's consider this code, to be sure of what we are doing :

final VideoBackgroundTextureInfo texconfig = mRenderer.getVideoBackgroundTextureInfo();
final Vec2I size = mRenderer.getVideoBackgroundConfig().getSize();
texconfig.setImageSize(size);

Now, to render it in a custom rendering engine, I need to set a plane/quad, to set the UV coordinates on it, and to place it front to the rendering Camera.

I made a quad, sizing the videoBackgroundConfig size, and set the UV to the example computation (VideoBackgroundTextureInfo img size / texture size)

The quad is centered in (0,0) and the rendering camera size the screen size, so the cropping is expected to be done.

I also compute this 

 VideoBackgroundConfig config = Renderer.getInstance().getVideoBackgroundConfig();
                DisplayMetrics metrics = new DisplayMetrics();
                activity.getWindowManager().getDefaultDisplay().getMetrics(metrics);
                uvPos = new Vec2I(
                        (((int) (metrics.widthPixels - config.getSize().getData()[0])) / (int) 2) + config.getPosition().getData()[0],
                        (((int) (metrics.heightPixels - config.getSize().getData()[1])) / (int) 2) + config.getPosition().getData()[1]
                        );

to move the camera by the half size.

 

But... at the end, the render is not good : on my phone it is not centered, on an other one it's good, but not well centered (so the picture displayed is not exactly the same than in yours examples)

I know this is just maths, and there is nothing really difficult but understanding all... but after a fews days on that I just become mad with my results.

Thanks a lot!

Video background configuration

February 20, 2014 - 11:42am #8

VideoBackgroundConfig specifies information primarily about the video size, while the VideoBackgoundTextureInfo contains information about the OpenGL texture size.

One thing to be aware of is the fact that the video texture has a size which is typically a bit larger than the actual video size; this boils down to the use of power-of-two dimensions for the video texture, such as 1024 x 512, while  the video dimensions are usually non-power-of-two, such as 640 x 480 for instance;

so, this means that the actual video image only fills a portion (a sub-region) of the video texture buffer, while the rest of it is just black.  As a consequence, the UV coordinates are computed so that only the video part is mapped on the background mesh, i.e. so that the black borders are not visible.

The sample code in the BackgroudTextureAccess reflects this fact, and if you look at the maths used, you should be able to understand how things work in detail. So, my suggestion is to look at that sample code and use the same maths implemented in there.

 

 

Video background configuration

February 20, 2014 - 8:39am #7

Hi, Still some issues to be fixed, so I need more informations please.

What about VideoBackgroundConfig vs VideoBackgroundTextureInfo ?

 

My goal is just to render on a well sized plane with the good UV coords the background texture.

Depending on the phone, I get not differents results I cannot explain.

And the documentation does not help so much, so... please tell me how the bind is done.

 

Thanks a lot!

Video background configuration

February 3, 2014 - 2:05am #6

Great to see you made such progress.

Now, concerning the Field of View, you can check this article, which explains it quite in detail;

https://developer.vuforia.com/resources/dev-guide/camera-parameters

I hope this will help you sort out the centering issue.

 

Video background configuration

February 3, 2014 - 1:49am #5

So, I set some new settings to manage the texture size as expected (on the paper, looks good this time)

The remaining issue is : the marker do not seems to be centered, so there is still a precision issue.

My first question is : how is the FOV of the Vuforia camera by default ? (Because I get the vuforia's camera position/rotation to apply on my engine, if the settigns are not the same, i will explain my issues )

 

Here are some screenshot to give you an idea of how it moves :

 

Thanks!

Video background configuration

January 31, 2014 - 8:19am #4

thanks !

I'll continue trying to understand it.

The main issue for me is because I use my own 3D engine rectangle to render the texture, binded by Vuforia.

I set the textcoords by mysef so I need to be sure of what Vofuria is doing on this texture size , ratio and texture support... 

Let's continue...

Video background configuration

January 31, 2014 - 6:51am #3

In Vuforia, video background rendering takes the following into account:

- screen width / height (your actual device screen resolution)

- VideoMode width / height: this represents the camera image (the video frame)

The VideoBackgroundConfig is basically used to setup the OpenGL viewport, so that the videobackground can cover the entire available screen.

Since the screen aspect ratio (screen width / screen height) is usually not the same as the VideoMode aspect ratio, some adjustment is needed for such viewport. 

So, the VideoBackgroundConfig width is set to either the screen width or to a value which is scaled up based on the aspect ratio (width/height) of the video mode; similarly, the same occurs to the config height. The idea of that code is to find a config width and height so that the video background texture will fit the screen; this also means that some of the video background texture may be outside the screen (i.e. the viewport origin may have negative coordinates, i.e.e off-screen, in the horizontal or vertical direction, so that the other dimension can fit the screen).

I know this may sound a bit complicated, but if you want to fully understand this mechanism, the best is to analyze the math in the code and see what is done in the Background Texture Access sample, where all the implementation details are revealed

Video background configuration

January 31, 2014 - 3:36am #2

And a little more question : 

 

Where starts the UV coordinates and in which direction please ? 

My 3D engine has it's rules but don't think they match vuforia.

 

Thanks.

Topic locked