Log in or register to post comments

Hand over 3D object - help

March 10, 2016 - 6:25am #1

Hi all,

I'm going crazy searching to make this thing. I want to show my hand over the 3D object fixed with Vuforia over a marker.

So, Vuforia handles the video and put the object with the marker reference. This is well documented and easy to do.

But what I can't do is the part to take the camera image (without 3D object) and put it over the scene. Then, I have a shader that does the chromakey work and delete the part with a particular color, so to create an hole and leave only the hand over the 3D object.


I hope that someone can help me. How can I take the camera image (without 3D object) and put over all?


Thanks in advance!

Hand over 3D object - help

June 9, 2020 - 6:09am #8



this looks a great!, I know its been few years and I am sure your techniques have improved! :)

Is there a way please to list the steps you ve done to accomplish this result?

I am a beginnner in unity and vuforia so if you could please give me basic instructions.

Many thanks in advance!


Hand over 3D object - help

March 17, 2017 - 5:09am #7


I try to do the same thing and  I think I'm pretty close, for what I have learn and ignore till now (ignorance is like an invisible wall, you see the other side but you cannot find the door to go there, especially where there is no door).

I haven' t accomplished it yet, cause i am not that good programmer but I have ideas to share, and I am pretty welcome to code solutions and development disquation..

What I think, is pretty close to the video with the "holding wooden cube" and an instance of a step furthermore...

In the video they use the green screen trick (alpha removal / chroma key). That is why the green fabric.

Both background and real cube are in colour tones of green.

In the unity setup inside the ARCamera prefab there is a plane (AARCamera --> Camera --> plane) that renders the webcam / smartphone camera input video (what you see of the real world, in your screen, is a simple video projection). What we can do is repeat this video, before the augmentation repeat this video with the alpha channel removed. What it's left is the hand.

Possible problems to take under consideration:

1) Scale problems. The hand might be bigger. <--> Maybe ortho camera is a solution.

2) Mark detection and alpha removal. There is a thin line between doing everything in tones of green and computer vision reads clear the mark identifier. <--> This is a try-fail-repeat procedure. What I think is creating / using a VUMARK in two green tones and check it in the vuforia target manager. What tones of green can be "deleted" needs further research. But if you achieve create the right VUMARK then you have a AR logo to print on stickers and stick them everywhere!

3) In real everyday use we cannot have a chroma key fabric with us, so to play. This is the real hard part. <--> What I am thinking, but still no results (not that I have done the above, just being pretty exited) is a hard code mixture of stencil shaders and chroma key removal. Somehow to repeat part of the background video input. The part that is being hidden by the augmentation (fingers / hands) with part of the real background that hopefully is chroma key masked by VUMARK. I haven't think any solution for the part of hand that cannot be masked by the VUMARK but I 'll come to a solution. Maybe skin recognition colour algorithms, but I have no idea if this do exists (just read it in a post). I also have found no tutorial / information / prefab and instructions on the use of stencil shaders.

4) All the above maybe easily run on the pc / unity editor, but it is not sure that our options will run on a mobile device. ex.: I have read somewhere that you maybe have to manually set smartphone to 24bit buffer something, for stensil shader to work, etc.

That's all till now,

Any ideas or implementations, even small parts of a single solution, please share....

Hand over 3D object - help

January 29, 2017 - 12:08am #6

Where can I get a video tutorial of doing this?



Hand over 3D object - help

March 13, 2016 - 8:17am #5

Good job!

Hand over 3D object - help

March 11, 2016 - 1:20pm #4

Nice!! You can upload one project for example? Thanks! :)

Hand over 3D object - help

March 11, 2016 - 12:05pm #3

Thanks a lot lot lot duncanx!

After hours and hours and many headache, I've done it! Now it's working!

Now need only some tweaks and enhance it, but I have a base to start my project.

For your doubts, I understand them. But for my project I know where hands will be, so the marker will be in a zone where there will be no hands on it.

Anyway, I found a video of some people with a working concept of markers+green screen. You can see it HERE.


Thanks again! You saved me! I have attached an image of my first proof of concept!

Image icon IMG_20160311_205621[2].png1.58 MB

Hand over 3D object - help

March 10, 2016 - 10:39am #2

I had an eyebrow raised as I read this...then I got to the part about the colorkey shader and realized that yes this is possible....but probably not with believable results.

You'll need to use multiple cameras. There are a number of articles about using multiple cameras (unity thing, not a vuforia thing...for sure google it). Basically though..look in a camera and see that "culling mask"...thats the set of layers that camera can "see". You can make more cameras and set them up to see different stuff. Then, also in the camera, set the "depth" which controls the order they will draw in. Then in your scene you could set different objects, ui, anything...to belong to the different layers that the cameras are culling. 

The magic ingrediant here is to realize that Vuforia has supplied you with the background render already. If you pop open the ARCamera, the last child is "BackgroundPlane". During runtime, this is where the background rendering can be accessed from. Use whatever method you fancy to get the visual data from that to the other camera...using your shader you would be ultimately getting an image of just your hand and then drawing that last.

The gotcha though is that when you put your hand into it...depending on where...you might block the imagetarget. And frankly I'm having trouble imagining how the background you're aiming the device at is both a greenscreen and an imagetarget with enough detail to be stable. 

Good luck!


Log in or register to post comments