Log in or register to post comments

2d tap detection

August 8, 2013 - 6:38am #1

My app is based on Video Playback (but it's already modified a lot). I'm implementing buttons on video (which are simple textures). I'm rendering buttons (textures) on X, Y and Z axis and it works as expected.

 

I have a problem with detecting tap action on those buttons. Based on tap detection that is done in Video Playback I did something similar.

 

Video Playback uses method "projectScreenPointToPlane" and then compares "intersection" with half of the size of trackable element for each side (making 3d problem a 2d problem). However, I think that it doesn't work correctly. Video targets are quite big so it's hard to debug but I noticed the problem when I created smaller tappable areas which are buttons.

 

As far as I understand intersection from method "projectScreenPointToPlane" should give X and Y from the CENTER of trackable right? From my testing X and Y values change a lot depending on the camera angle and trackable position. If I move my trackable to the center of screen than values are correct.

 

This is easy to reproduce (debug). Just put "NSLog(@"x: %f  y: %f", intersection.data[0], intersection.data[1]);" in method "tapInsideTargetWithID" in file EAGLView.mm. Open the app, move the camera to different angles and try to tap on a center of the trackable. As far as I understand it should give you values near to 0 (if you tap in the center) but it gives different values based on camera position.

 

I'm I doing something wrong and "projectScreenPointToPlane" works in a different way?

 

// Determine whether a screen tap is inside the target
- (int)tapInsideTargetWithID
{
    QCAR::Vec3F intersection, lineStart, lineEnd;
    // Get the current projection matrix
    QCAR::Matrix44F projectionMatrix = [[QCARControl getInstance] projectionMatrix];
    QCAR::Matrix44F inverseProjMatrix = SampleMath::Matrix44FInverse(projectionMatrix);
    CGRect rect = [self bounds];
    int touchInTarget = -1;
    
    // ----- Synchronise data access -----
    [dataLock lock];
    
    // The target returns as pose the centre of the trackable.  Thus its
    // dimensions go from -width / 2 to width / 2 and from -height / 2 to
    // height / 2.  The following if statement simply checks that the tap is
    // within this range
    for (int i = 0; i < NUM_VIDEO_TARGETS; ++i) {
        SampleMath::projectScreenPointToPlane(inverseProjMatrix, videoData[i].modelViewMatrix, rect.size.width, rect.size.height,
                                              QCAR::Vec2F(touchLocation_X, touchLocation_Y), QCAR::Vec3F(0, 0, 0), QCAR::Vec3F(0, 0, 1), intersection, lineStart, lineEnd);
        
        
        NSLog(@"x: %f  y: %f", intersection.data[0], intersection.data[1]);
        
        if ((intersection.data[0] >= -videoData[i].targetPositiveDimensions.data[0]) && (intersection.data[0] <= videoData[i].targetPositiveDimensions.data[0]) &&
            (intersection.data[1] >= -videoData[i].targetPositiveDimensions.data[1]) && (intersection.data[1] <= videoData[i].targetPositiveDimensions.data[1])) {
            // The tap is only valid if it is inside an active target
            if (YES == videoData[i].isActive) {
                touchInTarget = i;
                break;
            }
        }
    }

    [dataLock unlock];
    // ----- End synchronise data access -----
    
    return touchInTarget;
}

 

2d tap detection

August 13, 2013 - 8:27am #13

I only used this with QCARUtils (which is used across most of the samples) in version 2.0.7. Version 2.6.6. of VideoPlayback uses something different but no problem I will track the problem tomorrow and send you code for 2.6.6 :)

2d tap detection

August 13, 2013 - 7:55am #12

I am getting errors on these lines:

 

    CGSize size = [[QCARControl getInstance] viewSize];

    CGFloat contentScalingFactor = [[QCARControl getInstance] contentScalingFactor];

did you define methods for these that I can hang off QCARControl to try this out?

 

N

 

2d tap detection

August 13, 2013 - 7:08am #11

The impact is that method "projectScreenPointToPlane" was not giving intersection X and Y from the center of the trackable, it was dependend on the camera position. See my first post for more explanation. Photo that I attached also illustrates this issue:

 

https://developer.vuforia.com/sites/default/files/sample.jpg

 

If the method "projectScreenPointToPlane" would work correctly X on the center should be near 0.0 and it should be the same on the left and on the right (of course one should be "-" and one "+"). I think correct value for this trackable is around -60 for left and +60 for right.

2d tap detection

August 13, 2013 - 7:00am #10

Thanks for this.

Can I ask what problems were you seeing with the existing code in the existing video playback sample?

You say the code was a bit messed up, but what was the impact?

Also, presumably you made changes to both the methods supplied?

 

cheers

N

2d tap detection

August 13, 2013 - 6:06am #9

I just noticed that the newest source code in VideoPlayback uses something like QCARControl istead of QCARUtils. Here is the code for QCARUtils used like in Dominos sample.

 

Method "tapInsideTargetWithID" in EAGLView.m

- (int)tapInsideTargetWithID
{
    QCAR::Vec3F intersection, lineStart, lineEnd;
    QCAR::Matrix44F projectionMatrix = [QCARutils getInstance].projectionMatrix;
    QCAR::Matrix44F inverseProjMatrix = SampleMath::Matrix44FInverse(projectionMatrix);
    CGSize size = [[QCARutils getInstance] viewSize];
    CGFloat contentScalingFactor = [[QCARutils getInstance] contentScalingFactor];
    int touchInTarget = -1;
    
    // ----- Synchronise data access -----
    [dataLock lock];
    
    // The target returns as pose the centre of the trackable.  Thus its
    // dimensions go from -width / 2 to width / 2 and from -height / 2 to
    // height / 2.  The following if statement simply checks that the tap is
    // within this range
    for (int i = 0; i < NUM_VIDEO_TARGETS; ++i) {
        SampleMath::projectScreenPointToPlane(inverseProjMatrix, videoData[i].modelViewMatrix, contentScalingFactor, size.width, size.height,
                                              QCAR::Vec2F(touchLocation_X, touchLocation_Y), QCAR::Vec3F(0, 0, 0), QCAR::Vec3F(0, 0, 1), intersection, lineStart, lineEnd);
        
        if ((intersection.data[0] >= -videoData[i].targetPositiveDimensions.data[0]) && (intersection.data[0] <= videoData[i].targetPositiveDimensions.data[0]) &&
            (intersection.data[1] >= -videoData[i].targetPositiveDimensions.data[1]) && (intersection.data[1] <= videoData[i].targetPositiveDimensions.data[1])) {
            // The tap is only valid if it is inside an active target
            if (YES == videoData[i].isActive) {
                touchInTarget = i;
                break;
            }
        }
    }
    
    [dataLock unlock];
    // ----- End synchronise data access -----
    
    return touchInTarget;
}

 

 

Method "projectScreenPointToPlane" in SampleMath

SampleMath::projectScreenPointToPlane(QCAR::Matrix44F inverseProjMatrix, QCAR::Matrix44F modelViewMatrix,
                          float contentScalingFactor, float screenWidth, float screenHeight,
                          QCAR::Vec2F point, QCAR::Vec3F planeCenter, QCAR::Vec3F planeNormal,
                          QCAR::Vec3F &intersection, QCAR::Vec3F &lineStart, QCAR::Vec3F &lineEnd)
{
    // Window Coordinates to Normalized Device Coordinates
    QCAR::VideoBackgroundConfig config = QCAR::Renderer::getInstance().getVideoBackgroundConfig();
    
    float halfScreenWidth = screenHeight / 2.0f;
    float halfScreenHeight = screenWidth / 2.0f;
    
    float halfViewportWidth = config.mSize.data[0] / 2.0f;
    float halfViewportHeight = config.mSize.data[1] / 2.0f;
    
    float x = (contentScalingFactor * point.data[0] - halfScreenWidth) / halfViewportWidth;
    float y = (contentScalingFactor * point.data[1] - halfScreenHeight) / halfViewportHeight * -1;
    
    QCAR::Vec4F ndcNear(x, y, -1, 1);
    QCAR::Vec4F ndcFar(x, y, 1, 1);
    
    // Normalized Device Coordinates to Eye Coordinates
    QCAR::Vec4F pointOnNearPlane = Vec4FTransform(ndcNear, inverseProjMatrix);
    QCAR::Vec4F pointOnFarPlane = Vec4FTransform(ndcFar, inverseProjMatrix);
    pointOnNearPlane = Vec4FDiv(pointOnNearPlane, pointOnNearPlane.data[3]);
    pointOnFarPlane = Vec4FDiv(pointOnFarPlane, pointOnFarPlane.data[3]);
    
    // Eye Coordinates to Object Coordinates
    QCAR::Matrix44F inverseModelViewMatrix = Matrix44FInverse(modelViewMatrix);
        
    QCAR::Vec4F nearWorld = Vec4FTransform(pointOnNearPlane, inverseModelViewMatrix);
    QCAR::Vec4F farWorld = Vec4FTransform(pointOnFarPlane, inverseModelViewMatrix);
        
    lineStart = QCAR::Vec3F(nearWorld.data[0], nearWorld.data[1], nearWorld.data[2]);
    lineEnd = QCAR::Vec3F(farWorld.data[0], farWorld.data[1], farWorld.data[2]);
    linePlaneIntersection(lineStart, lineEnd, planeCenter, planeNormal, intersection);
}

 

2d tap detection

August 13, 2013 - 5:37am #8

Please by all means post this up and I will try to take a look :)

 

N

2d tap detection

August 13, 2013 - 5:15am #7

I tried to edit previous post but it did not work :)

 

I do not know what is your policy on fixing sample code but I can push some fixed code for review if you want to (I guess it's not on github or something?). If you manage it internally a ticket with reference for this thread should be more than enough to fix it.

2d tap detection

August 13, 2013 - 5:09am #6

Great to hear it's working for you :)

N

2d tap detection

August 13, 2013 - 5:06am #5

Thanks for your help. You were right about the method "projectScreenPointToPlane", it is simply messed in the VideoPlayback sample. I compared it line by line with Dominos and in VideoPlayback it takes wrong size parameters. It should use size from [[QCARUtilis getinstance] viewSize] and include [[QCARUtils getInstance] contentScalingFactor] in the calculations. I just don't know why in Dominos heigh and width is switched (see code below) but it seem to work.

 

float halfScreenWidth = qUtils.viewSize.height / 2.0f; // note use of height for width
float halfScreenHeight = qUtils.viewSize.width / 2.0f; // likewise

 

2d tap detection

August 9, 2013 - 2:25am #4

The Dominoes sample also uses projectScreenPointToPlane() in the handleTouches() method in Dominoes.mm so this would be a good starting point to experiment and learn. 

Within this method it also uses  checkIntersectionLine()  to check whether the line from  projectScreenPointToPlane()  intersects with the domino:

 

            bool intersection = checkIntersectionLine(domino->pickingTransform, lineStart, lineEnd);

 
Consequently you should be able to apply this to any 3D object, so long as it has a pickingTransform member and updates it just like the domino does using
 
updatePickingTransform(selectedDomino);

 

There is also further discussion about this topic in this thread:

https://developer.vuforia.com/forum/ios/getting-touch-event-3d-model

 

Most of what you want to do is already there, so you should be able to repurpose this without too much trouble.

HTH

N

2d tap detection

August 8, 2013 - 7:33am #3

Have you tested this functionin the Dominoes sample?  

 

As far as I know dominos uses 3d tap detection by using "checkIntersectionLine". Can I also make something similar here?

 

Do the X and Y values change as much, more or less in this sample compared to yours?

 

Yes mostly the same. Of course it depends on the trackable size but it's the same. I thought I did something wrong that's why I opened clean Video Playback sample and tested it there. See attached photo to see what I mean.

 

I think it's something with the Z axis.

BTW. Can I do tap detection on textures with "checkIntersectionLine" like in Dominos sample?

AttachmentSize
Image icon sample.jpg281.21 KB

2d tap detection

August 8, 2013 - 7:06am #2

From my testing X and Y values change a lot depending on the camera angle and trackable position

Have you tested this functionin the Dominoes sample?  

If so how is it different from this?  

Do the X and Y values change as much, more or less in this sample compared to yours?

 

N

Log in or register to post comments