Log in or register to post comments

Play remote video on image use calayer (sample code)

July 7, 2012 - 7:17am #56

I would like to share my recent work about playing video on trackable image, and ask some help.

Thank for vuforia team to release the sample code about how to play video using the vuforia sdk, they use avassetreaderoutput to render video pixel on the opengl texture.

However avassetreader can only work on local video file, if you want to play remote video, you need use native mpmovieplayer and into the fullscreen mode.

I try to solve this problem.

The idea is simple.

1.I don't use avassetreader to read the video pixel, instead I use AVPlayer and AVPlayerLayer which can play remote video file.

2.I need to convert opengl modelview matrix to CATransform3D so the AVPlayerLayer will attach on the trackable image.Thanks for Hammer on stackoverflow, he shared an example about how to make this.

3.I tried to render camera background use opengl and AVPlayerLayer at same time, but the performace is not good enough, so I use another calayer to render the camera background. There is a bug in 1.5.8 when getting camera frame, thank for andersfrank the problem was solved here.

I put my code on pastebin:

http://pastebin.com/kmeVZ7jy

 

I can play remote video now, but still have some problem need to solve:

1. I can't make the video flip up-side down.

2. it's also strange that I need to set m22 and m33 negative in scalingMatrix or the convertion between the modelview matrix and catransform3d is not correct. 

3. When I rotate my iphone, the attitude of AVPlayerLayer is not quiet well, may be the convertion is not good enough.

 

 

this one looks good.

this one is not.

 

I was stuck on this problem about 3 weeks. Please check the code and give me some feedback.

Thank you very much.

Play remote video on image use calayer (sample code)

July 10, 2012 - 1:58am #55

Good news here, I just fixed all the problem, for question 1 and 2, there are some column-major and row-major matrix convertion problem from GLKMultiply, I just replace all of them by ShaderUnits::multiplyMatrix. For question 3, you can fix it by set

cameraLayer.contentsGravity = kCAGravityResizeAspectFill;

This will fix the problem that the aspect ratio of the camera image is different with the iphone screen's.

I updated all my code on pastebin, good luck!

http://pastebin.com/kmeVZ7jy

Play remote video on image use calayer (sample code)

July 11, 2012 - 8:07am #54

Hi TimLai

This is great work - many thanks for posting it up.

I got the app working from the code you posted up and looks great.

Well done!

 

N

Play remote video on image use calayer (sample code)

July 11, 2012 - 8:51am #53

Dear NalinS,

You are welcome, I hope this code can help more people.

However the performace is not as good as the code using openGL to render the video, there is still much room to improvement. 

Play remote video on image use calayer (sample code)

July 12, 2012 - 11:06pm #52

Dear Tim,

I tried to integrate your code in Image Target Sample App of SDK but it is rendering errors such as QcarUtils.h not found.

I ahve also checked for the header paths and also for the frameworks included such as quartzcore but still unable to run the code.

Is it possible for you to share the sample app with me on

or

Please help me out as I am stuck in this for past 2 weeks.

Play remote video on image use calayer (sample code)

July 13, 2012 - 10:23pm #51

Hello Tim ,

I have implemented your .m code in place of AR_EAGLView.mm. 

Could you help me for the next step to be followed.

 

Play remote video on image use calayer (sample code)

July 14, 2012 - 8:24am #50

Dear junkieHead,

To make my code work, please follow the steps below:

1.Open the "ImageTragets" sample code project. (Maybe you should backup the original version first.)

2.In the project, create new class called ARCALView, the copy and paste my code to finish this class.

3.In the ARViewController class, replace all EAGLView to ARCALView.

4.Everything should work.

Good luck!

Play remote video on image use calayer (sample code)

July 15, 2012 - 12:03am #49

Thnx a lot Tim, the code is working now. But Whenever I use custom image target, it is displaying Data Set Load Failed.

Could you help me out in this

Thnx

Play remote video on image use calayer (sample code)

July 16, 2012 - 12:10am #48

Hi, junkieHead,

I think the problem is not come from CALayer, maybe you can post your question to  other topic and get useful answer.

Play remote video on image use calayer (sample code)

July 19, 2012 - 2:59am #47

Hi Tim,

Solved most of the issues, the one which i am left with is that i need to call a UIView nib file after i tap on the custom CALayer which i have overlayes.

I am using tapgesture recognizer in which i am trying to use the following:

 

-(void)handleButtonOverlayTap: (UITapGestureRecognizer*)recognizer{

    

    NSLog(@"ButtonOverlayTapped");

    webViewController *webViewController = [[webViewController alloc] initWithNibName:@"webViewController" bundle:nil];

    webViewController.modalTransitionStyle = UIModalTransitionStyleCrossDissolve ;

    [self presentModalViewController:webViewController animated:YES];

    

}

 

whenever i tap on the calayer , i get an exception of unrecognized selector sent to instance.

Plz help

 

 

Play remote video on image use calayer (sample code)

July 27, 2012 - 3:47am #46

Hi all,

Thanks for Tim's work. 

I am trying to create the avplayer when activeTrackable is found and call  

[player performSelectorOnMainThread:@selector(replaceCurrentItemWithPlayerItem:) withObject:item waitUntilDone:YES];

when other activeTrackable is found to play different remote videos for different images.

It works for first few times but exc_bad_access after few switch of videos.

Anyone has any idea on this?

Thanks for any help and advice.

 

Play remote video on image use calayer (sample code)

July 29, 2012 - 7:43am #45

Hey Guys I'm working on the similar problem, will appreciate if you can please share the sample code with me.

I'm Stuck on it for sometime. my email is:

Thanks all in advance !

Play remote video on image use calayer (sample code)

August 7, 2012 - 10:58am #44

anyone solved the BAD_ACCESS problem?

I tracked it down to renderFrameQCAR method and that is has do to with     cameraLayer.contents = cameraImage;

Play remote video on image use calayer (sample code)

August 7, 2012 - 12:03pm #43

it's solved.

Had to do with ithat I changed code to call initVideo after target was identified rather than initially. I split initialization and loading into 2 methods and called them at proper time, then BAD_ACCESS problem is gone.

I have not experiences any plane clipping and am really happy with the playback quality, even for remote video's.

BTW I have adjusted the video size so it matched the target size exactly; looks pretty cool.

Any thougths on whether this could work for Anroid too?

Thanks Tim for your contribution.

 

 

Play remote video on image use calayer (sample code)

August 10, 2012 - 1:41am #42

Hi Tim,

I am facing some issues with the video play code.

Its working flawlessly when being tested on iPhone4 but when same code is tested on iPhone 4S, its through exec_bad exception after few seconds of tracking.

Do you have any idea why is this happening?

Play remote video on image use calayer (sample code)

August 10, 2012 - 1:58am #41

junkieHead,

Sorry I don't have iPhone4S, do you have any information about which function cause this exception?

Play remote video on image use calayer (sample code)

August 10, 2012 - 10:46am #40

I would like to switch to another viewController from within the ARCalView. It works fine from the OverlayViewController but I want to do it without userinteraction, e.g. when the movie is done I want to switch to another view.

I have tried to call method back on the overlayViewController but I end up with an error like

Application tried to present modally an active controlle

Any ideas on how to this correctly?

 

 

Play remote video on image use calayer (sample code)

September 13, 2012 - 1:25am #39

Note that ARCALView is a subclass of UIView and not UIViewCOntroller . So, if you try to present another viewcontroller modally then it will throw exception. You should add your new view as a subview of ARCALView. That should sove your issue.

Play remote video on image use calayer (sample code)

September 13, 2012 - 1:25am #38

Note that ARCALView is a subclass of UIView and not UIViewCOntroller . So, if you try to present another viewcontroller modally then it will throw exception. You should add your new view as a subview of ARCALView. That should sove your issue.

Hi Tim, Would you please send

January 7, 2013 - 12:46am #37

Hi Tim,

Would you please send the source code to the below address for this query,so I can search how to resolve it.

gunsel5537@yahoo.com.cn

Daniel Cho

Play remote video on image use calayer (sample code)

January 23, 2013 - 6:38am #36

you sir, are a genious!

i had to adopt the code a little bit to get it working with Unity 4 and Vuforia 2.0.30 (i attached the code parts), but it's working!

thanks a lot!

 

- (void)render {
...    
    
    if (state.getNumTrackableResults() == 0 || videoInitialized == NO) {
        [player pause];
        hideVideo = YES;
        QCAR::Renderer::getInstance().end();
        return;
    }
... 
    // Get the trackable
    const QCAR::TrackableResult* result = state.getTrackableResult(0);
    QCAR::Matrix44F modelViewMatrix = QCAR::Tool::convertPose2GLMatrix(result->getPose());
...
}

...
- (void) postInitQCAR {
    // These two calls to setHint tell QCAR to split work over multiple
    // frames.  Depending on your requirements you can opt to omit these.

// Not sure what this is. doesn't work with vuforia 2.0.30 anymore
//    QCAR::setHint(QCAR::HINT_IMAGE_TARGET_MULTI_FRAME_ENABLED, 1);
//    QCAR::setHint(QCAR::HINT_IMAGE_TARGET_MILLISECONDS_PER_MULTI_FRAME, 40);
    
    // Here we could also make a QCAR::setHint call to set the maximum
    // number of simultaneous targets
    // QCAR::setHint(QCAR::HINT_MAX_SIMULTANEOUS_IMAGE_TARGETS, 2);
}

Play remote video on image use calayer (sample code)

January 23, 2013 - 7:49pm #35

Hi, atti,

 I am glad that my code can help you.

I also use my code on my iphone app -- vclip.

You can download it here. We also uploaded a demo video on youtube.

If you want to test this app you can go to our website and scan the postcard on second slider.

I use avplayerlayer to play http live streaming video with sdk 1.5.9.

I am also interested on how the sdk 2.0 can make my app more interesting.

If I have time I will try  sdk 2.0.

Play remote video on image use calayer (sample code)

January 31, 2013 - 10:01pm #34

Hi Tim,

 

I am running into exception while using the code. Can u help me by posting the code on

. Thanks much.

Play remote video on image use calayer (sample code)

March 2, 2013 - 9:00am #33

Hi, Nice work both,  Could you please explain to me in detail, how did you implement on unity vuforia? thanks you in advance :)

Play remote video on image use calayer (sample code)

March 4, 2013 - 12:45am #32

hi jpatinop,

just download the code and read through this thread. everythings has been explained.

Play remote video on image use calayer (sample code)

March 12, 2013 - 4:16am #31

Hi TimLai

Would you please send the source code to the below address for this query,so I can search how to resolve it.

Play remote video on image use calayer (sample code)

March 22, 2013 - 3:42am #30

hi Tim, We found your code is useful for our project but it seems not compatiable with SDK 2.0, have you update the sample with sdk 2.0?

Thank you very much

 

Update 1: Seems name it as .mm instead or .m will help. Thank you

Update 2: after few sec of tracking,  we got the EXC_BAD_ACCESS at [CATransaction commit]; Any idea?

Play remote video on image use calayer (sample code)

May 8, 2013 - 9:14am #29

 

Hi,

I tried to integrate your code in Image Target Sample App of SDK 1.5.9 but we got the EXC_BAD_ACCESS at [CATransaction commit];

anyone solved the BAD_ACCESS problem? Pls share the changes in code thanks in advance.

 

Thanks,

P.Uthaman

 

 

 

 

 

 

.

 

Play remote video on image use calayer (sample code)

May 13, 2013 - 3:28pm #28

Firstly TimLai, Great Code!! It has really helped me out!

I am having a bit of trouble with the position of the video on the trackable.

Does anyone know how I can bind the position, scale, skew  of the video to the position, scale and skew of the trackable?

I would really appreciate any suggestions! TIA!

Play remote video on image use calayer (sample code)

May 14, 2013 - 11:17am #27

It actually requires a lot of code and management to make the correct sizing at the rigth time to avoid flickering and resizing.Also, you need to considder if you want to keep the aspect ratio and if you want to scale to height or width.

In our App (https://itunes.apple.com/us/app/spiked-reality/id587840044?mt=8) you can see how it can work. The App streams remote video using HLS (adaptive streaming, required by Apple for larger video clips) and projects it into the target image depending on some server setting (eg where to place the video relative to the image, how to align to width and height). Try out the Sony Channel and get target images from Spiderman: https://s3-eu-west-1.amazonaws.com/spikedrealityimage/1000/1000_00001.jpg or https://s3-eu-west-1.amazonaws.com/spikedrealityimage/1000/1000_00003.jpg.

 

 

 

 

Play remote video on image use calayer (sample code)

May 15, 2013 - 10:17am #26

tgotzsche, I am glad to hear that you got it to work!

I have managed to get the video to display and it scales, however it does not seem to apply the proportions of the target image.

The biggest problem I am facing is that the video being displayed is not at the same Z-point as the target image.

Would you be able to post some example code so that others could learn?

It would be greatly appreciated! :)

Play remote video on image use calayer (sample code)

May 16, 2013 - 1:22am #25

Hi Tim,

I am studying on the same topic now, would you please send your code to me, that would be a great help, thanks!

my Email:

Play remote video on image use calayer (sample code)

July 4, 2013 - 5:09am #24

TimLai wrote:

Dear junkieHead,

To make my code work, please follow the steps below:

1.Open the "ImageTragets" sample code project. (Maybe you should backup the original version first.)

2.In the project, create new class called ARCALView, the copy and paste my code to finish this class.

3.In the ARViewController class, replace all EAGLView to ARCALView.

4.Everything should work.

Good luck!

Many thanks for figuring ou the problems, but I cannot pass through second step. I created ARCALview class (subclass of UIView) and placed it in the same folder as EAGLView was already placed. It produced tons of errors about parsing issues, like QCAR::DataSet. Another is targetType class which wasn't in project before, so what it should be... can anyone lend a hand so I could even test TimLai's solution?

edit: without adding ARCALView the ImagetArget sample is working properly (I used sample for ios sdk 2.5.7)

edit2: I changed ARCALView.m to ARCALView.mm which made the difference, but still have some bugs

QCAR::Matrix44F modelViewMatrix = QCAR::Tool::convertPose2GLMatrix(trackable->getPose());

is not finding getPose, and QCAR hints are not detected...

Play remote video on image use calayer (sample code)

July 4, 2013 - 6:48am #23

I tried to fix getPose with code snippet from EAGLView so it could build and QCAR hints with some hints from EAGLview as well, but ended up with:

2013-07-04 15:47:32.422 ImageTarget[1705:907] APPSTATUS_INITED

2013-07-04 15:47:32.481 ImageTarget[1705:907] APPSTATUS_CAMERA_RUNNING

2013-07-04 15:47:32.495 ImageTarget[1705:907] DEBUG/AR(1705) UIView has CAEAGLLayer class

2013-07-04 15:47:32.496 ImageTarget[1705:907] DEBUG/AR(1705) UIView responds to selector renderFrameQCAR

2013-07-04 15:47:34.175 ImageTarget[1705:907] *** -[AVCaptureVideoDataOutput setVideoSettings:] - videoSettings dictionary contains one or more unsupported (ignored) keys: (

    Width,

    AVVideoScalingModeKey,

    Height

)

2013-07-04 15:47:34.343 ImageTarget[1705:907] ARParentVC: dissappeared

2013-07-04 15:47:34.345 ImageTarget[1705:907] ARVC: viewDidDisappear

2013-07-04 15:47:34.346 ImageTarget[1705:907] QCARutils onPause()

2013-07-04 15:47:34.347 ImageTarget[1705:907] APPSTATUS_CAMERA_STOPPED

2013-07-04 15:47:34.529 ImageTarget[1705:907] -[ARCALView finishOpenGLESCommands]: unrecognized selector sent to instance 0x1e58e8b0

2013-07-04 15:47:34.531 ImageTarget[1705:907] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[ARCALView finishOpenGLESCommands]: unrecognized selector sent to instance 0x1e58e8b0'

So still solution is not working for me at all :/

Play remote video on image use calayer (sample code)

July 8, 2013 - 4:38am #22

bump - anybody?

Play remote video on image use calayer (sample code)

July 8, 2013 - 5:24am #21

I think you may need to contact the author of the code directly as it was developed outside.

 

N

Play remote video on image use calayer (sample code)

July 17, 2013 - 10:34pm #20

Dear all,

I was busy on my PhD degree last two month, so I didn't have time to answer any question from all of you.

I don't  use vuforia for a long time so I am sorry I still cann't answer your question.

Tim

Play remote video on image use calayer (sample code)

August 9, 2013 - 3:07am #19

I have just come across this and hopefully will be implimenting it in combination with the Cloud recognition to play the videos on the fly, somehting which many people seem to be asking questions about. 

In order to get Tim's example working with the newer version of QCAR you need to ammend the following lines:

 

if (state.getNumTrackableResults() == 0 || videoInitialized == NO) {
    [player pause];
    hideVideo = YES;
    QCAR::Renderer::getInstance().end();
    return;
}
    
hideVideo = NO;
if (player.rate == 0) {
    [player play];
}
    
    
// Get the trackable
const QCAR::TrackableResult* trackable = state.getTrackableResult(0);
QCAR::Matrix44F modelViewMatrix = QCAR::Tool::convertPose2GLMatrix(trackable->getPose());

You can now only call getPose() on a trackableResult and not on a Trackable. There is a lot more optimisation to be done on this as it crashes the app when the home button is pressed due to not having the cleanup code which EAGLView does have. Anyway hope this helps.

Play remote video on image use calayer (sample code)

August 9, 2013 - 3:31am #18

Thanks very much for sharing :)

N

Play remote video on image use calayer (sample code)

September 18, 2013 - 7:41am #17

Hi there,

do you think it's possible to implement that with Unity? If I change manually the iOS code with xcode, I'll have to do this each time...

 

Thanks !

Play remote video on image use calayer (sample code)

September 18, 2013 - 7:58am #16

I don't have that particular project opened at the moment, but usually working with native code in Unity works like this:

Within the Assets folder you have one or multiple "Plugins" folder(s) that contain libraries or source code files (.h .m .mm ...).

When you build your project Unity creates the XCode project and copies those files within Plugins over and includes them.

So what you should do is find the source files within you Assets folder and manipulate them there instead of within the XCode project, so that the changes are permanent.

hth

Play remote video on image use calayer (sample code)

September 18, 2013 - 8:14am #15

Hi atti,

Thanks for your answer, indeed that solves the problem of having to do this every time, but in this case the native sources are packaged in .a files. I guess we will need someone at Qualcomm to implement those changes and recompile the .a files for use with Unity.

Play remote video on image use calayer (sample code)

September 19, 2013 - 6:37am #14

I figured out how to run the sample natively (iOS) with the code from TimLai using Vuforia 2.6.x (latest version - 2013/09/19)

I share with you the xCode project, note that in the line 164 from ARCALView.mm script you must replace the url by yours.

 

ImageTarget sample is used rather than VideoPlayback (I suppose that this sample were not created when TimLai posted this fix).

I changed the extension of the ARCALView file to mm (fixed a lot of errors due to C code).

I changed the code to work with vuforia 2.x (a lot of fixes, due to major changes on the API).

 

During my tests I tried first to create a server using Python (on a mac), the video were not readable, I don't really know why.

I tried to use a Mamp server, and it worked. The streaming is smooth (local networking) and the framerate is decent (over 30fps, tested on iPod touch 5).

 

It would be great to have this fix on the Unity extension. I don't really know where to start, it seems that a lot of work done by the ARCALView is also done into the VideoPlaybackBehaviour class in Unity, or other classes. The first job will be to identify  real modifications (things that TimLai explained) and apply them into the VideoPlayback sample. If someone wants to work on that, it will be really appreciated !

 

But still, I hope this will help someone !

 

Thanks, Loic.

AttachmentSize
Package icon VideoPlaybackStreamIos.zip765.83 KB

Play remote video on image use calayer (sample code)

December 27, 2013 - 2:37am #13

Hi loic_illogika,

                 super xcode project using imagesample into videoplayback as soon as possible add particular image target find chage videourl like every target image having particular video url.

 

 

Play remote video on image use calayer (sample code)

March 31, 2014 - 6:53am #12

Hello!

Thanks for sharing the info! I used TimLai's and your code and got it working. I am using SDK 2.8.7 with Unity3D 4.2.1f and don't know much, rather very little about XCode. As the libraries have changed a couple of times from SDK 2.0.3 to 2.8.7, I am not able to get the code working in SDK 2.8.7 videoplayback sample.

XCode throws errors about missing *.h and *.m files like ShaderUtils, QCARutils, etc. and goes on.

Could you guide me a little about the same. It would be great if you could tell me something about bridging to Unity3D also (as mentioned, I know very little about XCode).

Any help is much appreciated. Thanks in advance! :)

Play remote video on image use calayer (sample code)

July 15, 2014 - 4:22pm #11

I am looking at using this sample code now; were you able to get it to run with recent versions of Vuforia?

Play remote video on image use calayer (sample code)

August 11, 2014 - 8:54am #10

Did anyone worked on integrating this as a Unity Plugin, in order to get to choose the URL on the fly?

Play remote video on image use calayer (sample code)

September 22, 2014 - 6:38pm #9

may i ask you why i can't open the download url, i am in china ,but i have set the VPN. and i can login the facebook.

Play remote video on image use calayer (sample code)

December 7, 2014 - 3:59am #8

I downloaded the code, which aparrently does not work, as it concerns version 2.xx of the vuforia SDK, but now we have version 3.xx.

In the end, has anyone managed to do it?

Play remote video on image use calayer (sample code)

August 11, 2015 - 2:49am #7

你好,我也是在做这个。

就是要扫描图片然后下载服务器端的视频再播放。

我现在遇到问题,没法播放服务器端的视频。但在安卓端上很容易就实现了

请问你可以指导指导么?谢谢

Play remote video on image use calayer (sample code)

August 11, 2015 - 2:49am #6

你好,我也是在做这个。

就是要扫描图片然后下载服务器端的视频再播放。

我现在遇到问题,没法播放服务器端的视频。但在安卓端上很容易就实现了

请问你可以指导指导么?谢谢

Log in or register to post comments