Log in or register to post comments

Cool Video Texture effect

January 16, 2012 - 7:29pm #1

I made a special effect, similar as displayed in this video:
at time 2:55

It uses the video texture provided by QCAR and a special shader I have written. Here is the original from the video:

And here is my version:

Note that my version isn't as dynamic as the original as you can only change the size and shape of the hole but not move it around and it also doesn't have collision detection. It can be used as a starting point for more complex effects though.

This is how it works:
The UV coordinates of the mesh are modified so that the video texture is effectively "projected" straight onto the geometry. Then the geometry is deformed, "stretching" the video texture with it. For the UV coordinates, the vertex position in screen space from the un-deformed mesh is used. The vertex position and UV position use a different range (-1 to 1 and 0 to 1 respectively) so this has to be taken into account. Also the video texture is mirrored and flipped with reference to a standard Unity3d plane mesh and it's standard UV coordinates, so this has to be taken into account as well. Once the correct UV coordinates are calculated, the vertex can be displaced in the shader and the video texture will be stretched with it automatically.

The effect is included in my project here:

Cool Video Texture effect

February 16, 2013 - 5:16am #29

I added another special effect which makes use of the video texture to UCS.

It is similar to the effect in this video at 0:47

From the UCS manual:

Texture Patching

This special effect takes a snapshot of the video texture and applies it to any flat plane. Use it if your object has to move around with the video texture stuck to it at the time of the snapshot. You can use this for example to make a piece of a wall fall to the ground with its original real world texture still being visible.

Before the texture freeze is triggered, the object is invisible. Once the video texture for the object is frozen, the object edges will not blend in perfectly with the background, unlike with the object deforming effect. This is no problem because after the texture is frozen, the object is moved around and does not need to blend in with the background. The object will not blend in as good as the object deforming effect because the object will not have the same moving image noise as the rest of the video texture and will not react to real world lighting changes. The texture patching effect requires a flat plane and object deforming is not supported with this effect as it does not make sense to move a deformed or non-flat object to another location with the video texture stuck to it.

Note: the game object which this shader is attached to must have a uniform scale such as 1, 1, 1, or 2, 2, 2, but not for example 1, 0.5, 1. If you want to create a different shape you have to change the mesh itself.

UCS supports to different methods of achieving this effect:

-RenderTexture is the fastest method but requires Unity IOS and/or Android Pro.

-ReadPixels does not require Unity Pro but it is slow and causes a visible frame stutter when used.

To enable one method or the other, open SpecialEffect.cs and enable or disable this line:


If enabled, UCS uses RenderTexture. If disabled (removed or commented out), UCS uses ReadPixels.

Note: Render texture in the Unity Editor only works if PC/MAC build target is selected and active.

Example: VidTexPatch scene (press "Play Animation" button in Game or Geometry mode).

Material: Project window -> Materials -> VidTexPatch

Re: Cool Video Texture effect

March 19, 2012 - 4:58pm #28

Good to hear that it works. Keep up the good work :-)

Re: Cool Video Texture effect

March 19, 2012 - 7:50am #27

I followed carefully the UCS project and exported the right prefabs to include in my project and drop them into the EffectScript. It works perfectly! This is really awesome thanks a lot for that. I will play around and experiment various stuff with it, I already added the paint vertice script from the procedural example onto the camera the effect is so cool. I will try with an animated mesh now. Will keep you posted.

Re: Cool Video Texture effect

March 19, 2012 - 4:10am #26

Here is the script. All I did was delete everything from the MarkerRecorder.cs script which you don't need. Don't forget to rename it to cs instead of txt.

Again, this is not a shoot-and-forget script. Not designed for non-programmers. I wish I could make the script more simple but unfortunately I can't. The video texture Vuforia provides is very non-standard (for the sake of speed) and requires some additional code to extract the variables needed. You cannot add this script to every prefab which uses the plane effect shader. Instead you have to add this script to the camera and drop the plane effect prefab onto this script in the editor. I could design the script so that you attach it to every plane effect prefab instead but then you will need inter script communication to extract the necessary variables, which can become messy as well.

If you can't get it to work, look at my UCS project again and see how it is set up. It uses the same system.

Re: Cool Video Texture effect

March 18, 2012 - 7:15pm #25

Sorry if it wasn't clear, I am talking about the last shader working based on the mesh I gave you. Here is what I got:

The ScaleFacXa, ScaleFacYa etc are set to 0 (these parameters are for setting the UV coordinates right?). Like you said these values are not to be modified. Yes it would be cool to have just the script that sets these values correctly and nothing else.

Sorry for the confusion.

Re: Cool Video Texture effect

March 18, 2012 - 4:22pm #24

[QUOTE" />That would be awesome if you could send me a script only for setting up the correct values for the plane shader effect.[/QUOTE" />

I am not sure which shader you are talking about. The first shader I made will not work for pre-deformed mesh. The second shader, like I said before, does not have any values to be modified. Maybe me talking about modifying UV coordinates gave you that impression, but UV coordinates are not to be modified.
Or do you want to have a script which just sets the UV coordinates and nothing else? Please elaborate.

Re: Cool Video Texture effect

March 18, 2012 - 9:16am #23

Indeed I am not programmer however I am pretty familiar with visual programming. I used to work wth virtools back in time but now I am using Antares Vizio:
Really powefull tool, perfectly adapted for profiles like me (Interaction Designer). We made this little demo using only Antares Vizio and String: http://www.visionaries777.com/arspaceship/

I would really like to experiment various visual effects based on the video for an AR game. That is why I asked you if it was possible to make it based on a deformed mesh, because I think it could be more user friendly and flexible (in my case at least) to experiment video effects. Just like animating the mesh I gave you would be pretty easy in a 3D software...

That would be awesome if you could send me a script only for setting up the correct values for the plane shader effect.

Thanks a lot again.

Re: Cool Video Texture effect

March 18, 2012 - 7:36am #22

The shader variables are indeed set in the MarkerRecorder.cs script and depend on both the screen resolution and the picture-in-picture size of the video texture supplied by Vuforia. These variables are used to get the UV texture mapping right.

The variables are set in a function called "SetPlaneEffectShader()" but this will be renamed in the next version. That function is in turn called from the CheckVuforiaVidInit() function, which is called from the Update() function.

Again, that function only sets some variables to get the UV mapping right so the video texture is correctly mapped across the geometry. It has nothing to with morphing or animation.

Also, you mentioned you are not a programmer. I suggest you get to grips with it. Writing games without programming is not really an option unless you want to do something really simple. The Unified Coordinate System or the shader(s) that come with it are not designed for non-programmers and you will run into many obstacles very quickly.

Re: Cool Video Texture effect

March 18, 2012 - 7:22am #21

Cool! I tried it in Unity. Seems to work indeed when I adjust the values in the Shader settings I can see that that the mapping is right now. However I can read at the beginning of your shader: "do not change these values manually. The values will be set at runtime". I guess it is in the Marker recorder script, depending of the device and the resolution of the screen...
Could you isolate the part of the script setting up the shader values at runtime?
Because I would like to incorporate this shader in a simple AR project.

Thanks a lot, looking forward to try it.

Re: Cool Video Texture effect

March 18, 2012 - 5:03am #20

I made it to work. The solution was super simple in the end. Oh well.
Here is the shader:

This version is the same as PlaneEffect.shader, except that it uses the mesh itself
instead of a math function to deform the mesh. It assumes that the mesh "original" shape
is a flat plane where all y coordinates are 0. 

Shader "Custom/PlaneEffect2" {
	Properties {
		_MainTex("Texture", 2D) = "white" { }	
		//Note that these variables will show up in the Unity
		//editor, but but do not change these values manually.
		//The values will be set at runtime. If the shape of the
		//morph is to be changed, change "holeSize" only (in the
		//cs script) and recalculate the remaining variables.  
		_ScaleFacXa("ScaleFacXa", Float) = 0
		_ScaleFacYa("ScaleFacYa", Float) = 0		
		_ScaleFacXb("ScaleFacXb", Float) = 0
		_ScaleFacYb("ScaleFacYb", Float) = 0	
	Cull Back
	#pragma vertex vert
	#pragma fragment frag
	#include "UnityCG.cginc"
	sampler2D _MainTex;
	float _ScaleFacXa;
	float _ScaleFacYa;
	float _ScaleFacXb;
	float _ScaleFacYb;
	struct v2f{
		float4  pos : SV_POSITION;
		float2  uv : TEXCOORD0;
	float4 _MainTex_ST;
	v2f vert(appdata_base v){
		v2f o;
		float2 screenSpacePos;
		float yOffset;
		float magnitude;
		float4 clipPos;
		//get the y vertex position and store it
		yOffset = v.vertex.y;
		//now set the y vertex to 0
		v.vertex.y = 0.0f;
		//convert position from world space to clip space
		clipPos = mul(UNITY_MATRIX_MVP, v.vertex);
		//Convert position from clip space to screen space.
		//Screen space has range x=-1 to x=1
		screenSpacePos.x = clipPos.x / clipPos.w;
		screenSpacePos.y = clipPos.y / clipPos.w;
		//If the video texture would not be flipped,
		//the screeen space range (-1 to 1) has to be converted to
		//the UV range 0 to 1 using this formula:
		//o.uv.x = (0.5f*screenSpacePos.x) + 0.5f;
		//o.uv.y = (0.5f*screenSpacePos.y) + 0.5f;
		//However, due to the fact that both the Vuforia and String
		//video texture is flipped, we have to take this into account. 
		//Also, the video texture can be clipped, so we have to take this 
		//into account as well.
		o.uv.x = (_ScaleFacXa * screenSpacePos.x) + _ScaleFacXb;
		o.uv.y = -(_ScaleFacYa * screenSpacePos.y) + _ScaleFacYb;	

		//restore the original y offset.
		v.vertex.y = yOffset;
		o.pos = mul(UNITY_MATRIX_MVP, v.vertex);		
		return o;
	half4 frag(v2f i) : COLOR{
		half4 texcol = tex2D(_MainTex, i.uv);
		return texcol;

I will include it in the next release with a proper material setup. For now you can replace the original planeEffect shader with this one but make sure you do not call the SetMorph function on the gameObject containing this shader. This is because there is no receiving end for it and it in the shader and it will make the program run very slowly.

Also make sure you have a decent amount of surface features in order to see the effect properly.

The shader assumes that the "original" mesh is flat (all y coordinates are 0) but this is the case nearly all the time anyway, like with the mesh you gave me. The original is flat and then all (or some) y coordinates are displaced to create a deformed mesh.

It should also work with an animated mesh although I did not test that.

Re: Cool Video Texture effect

March 17, 2012 - 10:02pm #19

I will give a try to the Unified Coordinate system new version on Monday.
Wow that would be awesome if you could make it work, here is the mesh I used: http://www.visionaries777.com/TEMP/qcar/Plane_effect.FBX

Let me know if it works ;)

Re: Cool Video Texture effect

March 17, 2012 - 8:38pm #18

I updated the Unified Coordinate system. See that forum post to download it.

And flasorne, can you post me that deformed mesh of yours or explain how to make that? I have an idea how to fix it and I need a mesh to try it on.

Re: Cool Video Texture effect

March 16, 2012 - 4:53pm #17

You could do that using a height map. I tried to do that some time ago but I got stuck on the fact that not all devices support texture fetch in a vertex shader. An approach which gets the vertex position before and after the deformation for each separate vertex would be very slow.

In any case, you cannot just remove the math function which deforms the mesh in the current implementation. Without the math deformation, the texture is "projected" as it were on top of the mesh given to it by Unity3d. So that alone won't give you the desired effect. What you want to do is project the texture on a flat mesh plane, and then deform it, pulling the texture with it. This is not the same as using the Unity3d projector function as that will do the same as my vertex shader without the math deformation function.

I have some ideas how to do it but at the moment I can't get anything to work. To be continued.

Re: Cool Video Texture effect

March 16, 2012 - 2:56am #16

That's cool if it is just a bug to fix in your code, will wait patiently for the next release ;)
I wanted to share with you another approach for deforming the video. As I am not a coder it will be complicate for me to write down my own shaders and come up with my own video effect. So I was thinking in theory it should be possible to map the video feed using your shader onto a deformed plane.

Here is a example:

I would like to be able to map the video on this deformed plane. And may be having it animated later.

So I looked a bit in your shader and removed the hole effect but keeping the correct mapping relative to the camera position:

When I apply it to my mesh I obtain this:

The mapping is only flat and doesn't follow the mesh.
But you can see that the deformations on the mapping are right.
Is there anyway to fix this by modifying your shader?

Thanks a lot for your feedbacks.

Re: Cool Video Texture effect

March 15, 2012 - 5:01am #15

The visible seam is caused by a bug in my code. This is what I was discussing with Kim in this thread. I have found the problem and already fixed it. I will publish it in my next release, which will be soon.

You can scale the size of the special effect plane to suit your needs. Be sure to adjust the worm hole size related variables accordingly so the hole is not too big for the plane.

Re: Cool Video Texture effect

March 15, 2012 - 2:21am #14

Alright my bad sorry for that, just noticed that you had a section Setup QCAR in your documentation... I just followed the Prerequisites and imported your package. Off course now it works like a charm. Really awesome system, thanks for making it happend and sharing it. It looks like the PS Vita is using a similar system:

The plane deformation effect is now working in the UCS project. I will try now to implement the effect only in another project based on a single imagemarker.

I noticed one detail, it looks like the size of the plane is too small and doesn't overlap properly the video background. Is it something adjustable with the size of the plane (using the createplane.cs)?
[IMG" />http://www.visionaries777.com/TEMP/qcar/IMG_0381.PNG[/IMG" />

Re: Cool Video Texture effect

March 15, 2012 - 1:03am #13

-The video is supposed to be rendered at the background as well and you should not see the yellow background, so something is still not right with your video plane and orthographic camera settings. Again, just forget about setting up a project manually for the moment. Use my Unified Coordinate project instead. It contains my video morphing effect as well. Can you get that morphing effect to work in the UCS project? It is only visible in game mode by the way.


But sometimes I can see a cube moving very fast in random positions and orientation in space very weird.

Sounds like you didn't follow the installation steps in the manual EXACTLY. You will have to make some modifications to the "QCARBehaviour.cs" file and modifiy the script execution order as well.


I tried to set the world center mode to None and Auto but I got the same result.

the installation is NOT to be modified.

The installation process is to be followed exactly in the order in which it is described. Have a look again at the topic "Vuforia setup" and try again.


I don't see any cube on any marker even if I try to select a marker before or after.

In order to add geometry you will have to record a scene first, save it, go to geometry mode, select a marker, and click a button which adds geometry like "Add Cube". In that exact order. No point in selecting a marker after you add geometry. The reason you select a marker is to indicate where the geometry should appear.

Does that help?

I found out what the problem is. It indeed has to do with aspect ratio. On the Galaxy Tab 7, the screen is wider then the camera image texture. Because of this, the video screen is scaled down. The vertex shader expects the corner coordinates of the screen to line up exactly with the corner coordinates of the video texture. But this is not the case if the video screen scaled to anything but 1,1,1. I should be able to fix that.

But in the process, I thing I found a bug in the background texture example. When using an orthographic camera, the scale of the video screen is always set to the same scale, regardless the output of ShouldFitWidth().

Re: Cool Video Texture effect

March 15, 2012 - 12:30am #12

I have this message "Select Marker and keep in view" I tried but nothing happend:

Re: Cool Video Texture effect

March 14, 2012 - 10:42pm #11

Elecman, I have been able to make your shader work however I still have the yellow background from the original backgroundcamera. Is the video supposed to be displayed on the background as well?

Here is a screenshot my iphone:

I tried to set the world center mode to None and Auto but I got the same result.

I also had a look to your project. I bought vectrosity and UnifileBrowser. I managed to build the project on my phone. I followed your steps in the doc about recording and saving... until here everything seams to work fine. However as soon as I try to add geometry cube for instance I don't see any cube on any marker even if I try to select a marker before or after. But sometimes I can see a cube moving very fast in random positions and orientation in space very weird.

I am using vuforia1.5.9, unity3.5 and iphone4S.

Thanks for your help

Re: Cool Video Texture effect

March 14, 2012 - 6:14pm #10

flasorne, you cannot use my shader on any object. It will just "disappear" into the background as the geometry texture is the same as the video texture. You need to deform the geometry in the vertex shader to see any effect.

And about the yellow background, this is because the background camera is not pointing at the video texture plane which indeed is called NegativeGrayscaleVideo in the Vuforia example. Make sure you use landscape left orientation for the iphone and examine the transform of both the video plane and the camera to confirm the camera is pointing at the plane. Note that there are two cameras in the scene. One which renders the normal game content and the other one which just renders the video plane. They are hidden from each other using Depth.

Again, please look at my project I gave you the link for and also look carefully how the Vuforia example is set up. Pay close attention to which camera has which Depth and what the parameters of the two cameras are.

Re: Cool Video Texture effect

March 14, 2012 - 7:42am #9

So basically I have a Cube child of the imagetarget that I put in the augmented object field of your script.
Then I use the BackgroundCamera provided in the sample BackgroundTextureAccess. And the NegativeGrayscaleVideo as I asume is the video screen... But may be I am wrong.

But what I get once exported to my iPhone is that the video feed is displayed on the texture of the cube. And I get a yellow background from the BackgroundCamera :(

Thanks a lot for your help.

Re: Cool Video Texture effect

March 14, 2012 - 7:34am #8

@flasorne: Don't bother setting it up manually. There are various settings which are easy to get wrong. Download my project which includes the special effect here:
That should get you started.

@Kim: I tried the sample code, including the automatic mesh generation but the same problem still exists...

Re: Cool Video Texture effect

March 14, 2012 - 3:11am #7

Great Stuff Elecman! Really looking forward to play with it.
I am trying to make it work but I am not really sure how to set the things up in Unity.
Are you starting from the BackgroundTextureAccess sample? Could you show me a screenshot of your scene in Unity and your hierarchy?

Thanks a lot for your help.

Re: Cool Video Texture effect

March 13, 2012 - 5:32pm #6

I think this is the problem:

//We need to make sure that the horizontal size of the ortho camera has the same size as the horizontal
//size (x length) of the video plane. The formula for that is:
//ortho_cam_size = (video_gameObject_width/2) / (Screen.width/Screen.height)
Camera backCameraComponent = backgroundCamera.GetComponent<Camera>();
float monitorAspectRatio = (float) Screen.width / (float) Screen.height;
backCameraComponent.orthographicSize = defaultPlaneSize / monitorAspectRatio;

You don't always want to fit the camera image to the width of the screen. That works on most devices, but on some devices the aspect ratio of the camera is "wider" than that of the screen. In that case you need to stretch the camera image to fill the height of the screen.

See this code snippet from the VideoTextureBehaviour class included in the BTA sample:

private bool ShouldFitWidth()
    if (mScreenWidth > mScreenHeight)
        float height = mTextureInfo.imageSize.y * (mScreenWidth / (float)
        return (height >= mScreenHeight);
        float width = mTextureInfo.imageSize.y * (mScreenHeight / (float)
        return (width < mScreenWidth);

Let me know if this helps, if not I'll take another look.

- Kim

Re: Cool Video Texture effect

March 13, 2012 - 2:39am #5

I noticed that this effect works perfectly on the iPad 2 but on a Samsung Galaxy tab 7 (GT-P1000), the UV coordinates are slightly offset causing the texture of the effect plane to blend incorrectly with the background texture. This becomes apparent by a visible seam.

Kim, do you have any idea what might be causing this?

Re: Cool Video Texture effect

January 28, 2012 - 8:02am #4

Thanks, I appreciate the appreciation :)

Re: Cool Video Texture effect

January 27, 2012 - 10:37am #3

thanks elecman, the preview looks great.
i really appreciate you sharing your work! and i'll give it a try tomorrow.

Re: Cool Video Texture effect

January 20, 2012 - 6:02pm #2

I updated the attached Unity3d script. It is now compatible with String as well. The shader remains the same, no changes needed.

Log in or register to post comments