Log in or register to post comments

How to make Vuforia working with AVCaptureVideoPreviewLayer and SceneKit?

October 29, 2014 - 5:25pm #1
I am developing an Augmented Reality app using Vuforia sdk. I am trying to use AVCaptureVideoPreviewLayer and SceneKit for application rendering instead of raw OpenGL calls provided by Vuforia sample code.
 
I got the AVCaptureVideoPreviewLayer and SceneView working without Vuforia, i.e. I managed to draw 3D scene on top of camera video background. The code is at: https://github.com/lge88/scenekit-test0/blob/master/scenekit-test0/GameViewController.swift#L74-L85: 
 
    func initViews() {
        let rootView = self.view
        
        let scnView = createSceneView()
        let scene = createScene()
        scnView.scene = scene
        
        let videoView = createVideoView()
        
        rootView.addSubview(videoView)
        rootView.addSubview(scnView)
    }
 
The implementation can be summarized as:
 
 1. Create a UIView called videoView.
 2. Initialize an AVCaptureVideoPreviewLayer, and add it as a sublayer of videoView.
 3. Create a SCNView called scnView and initialize the scene o scnView.
 4. Add both videoView and scnView to the root UIView.
 
 
    initViews()
    animateScene()
    initControls()
    ARServer(size:viewFrame.size, done: initARCallback)
 
ARServer is a class that takes care of the Vuforia initialization, its implementation is taken from Vuforia ImageTargets sample code. The tracker is working, it can successfully track the targets of the sample dataset.
 
However the AVCaptureVideoPreviewLayer rendering doesn't work correctly, the area of the video rendering area is resized, and the video layer is not updating, it shows a static image captured when the tracker camera started. Here is how it looks from a ipad screenshot: https://github.com/lge88/scenekit-test0/blob/master/video-preview-layer-issue.png
lge
Joined:
March 8, 2013
Posts: 1
Top

How to make Vuforia working with AVCaptureVideoPreviewLayer and

February 27, 2016 - 12:36am #7

I wrote the article about how to integrate SceneKit into Vuforia('s OpenGL context). Hope it helps.

http://qiita.com/akira108/items/a743138fca532ee193fe

akira108
Joined:
December 2, 2015
Posts: 1
Top

How to make Vuforia working with AVCaptureVideoPreviewLayer and

April 21, 2015 - 10:47am #6

Hello, I had been able to render the scene kit over the OpenGL context used by vuforia. And I'm having problem with the camera. Did you can solve your problem?

 

I'm calculating the camera position and orientation, then I set the position of my scene camera and set the lookAt constrait. But the camera is not working fine.

 

 

dcorbatta
Joined:
March 16, 2015
Posts: 3
Top

How to make Vuforia working with AVCaptureVideoPreviewLayer and

March 31, 2015 - 8:03am #5

I'm trying the same but I don't see the scene over the video. Are you doing something else? What does your plane.dae have ?

dcorbatta
Joined:
March 16, 2015
Posts: 3
Top

How to make Vuforia working with AVCaptureVideoPreviewLayer and

March 31, 2015 - 7:30am #4

I am going off of the imagetargets example.... 

 self.context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];

 if (_context != [EAGLContext currentContext])

   [EAGLContext setCurrentContext:_context];

    

self.renderer = [SCNRenderer rendererWithContext:(__bridge void *)_context options:nil];

self.sceneModel = [SCNScene sceneNamed:@"art.scnassets/plane.dae"];

self.sceneCameraNode = [SCNNode node];

[_sceneModel.rootNode addChildNode:_sceneCameraNode];

_sceneCameraNode.camera = [SCNCamera camera];

[self.renderer setScene:_sceneModel];

 
The missing piece is the camera position. It should be possible to calculate it inside renderFrameQCAR from the pose matrix and modify sceneCameraNode but I have not any any success there. 
aaronpride
Joined:
January 3, 2012
Posts: 3
Top

How to make Vuforia working with AVCaptureVideoPreviewLayer and

March 31, 2015 - 6:42am #3

aaronpride: Could you share your code? I' trying to do the same but I get the error: scenekit: error, unable to create context. I also tried to add the scene view as an overlay view, but the EAGLView doesn't render the video.

 

dcorbatta
Joined:
March 16, 2015
Posts: 3
Top

How to make Vuforia working with AVCaptureVideoPreviewLayer and

March 25, 2015 - 8:04am #2

I was able to get image targets working, and render a Scene on top of the video output by setting the renderer up like this:

 self.renderer = [SCNRenderer rendererWithContext:(__bridge void *)_context options:nil];

I tried using the post describing how to calculate camera angle and position from modelview matrix, and got the scenekit camera to move roughly in relation to the target. I am missing something though, the scene is not aligned well on the target.  Seems like there is good potential for scenekit vuforia integration though, would appreciate a sample project to that end.

 

 

 

 

aaronpride
Joined:
January 3, 2012
Posts: 3
Top
Log in or register to post comments