One additional little detail is that you need to render the "plane with hole" before rendering anything else, so to write to the depth buffer before other geometry and thus obtaining the clipping effect.

And by the way: disabling Color Mask (i.e. disabling writing to color buffer) in OpenGL is done by:

glColorMask( false, false, false, false );

Thank you for your PM messages with details; so I'll share the answer to your problem here, hoping it can be useful to other developers as well:

So, the technique you need to use is based on rendering a plane to the "depth buffer" without rendering it to the "color buffer";

in practice, you need to create a geometry which is like a plane (a rectangle) with a rectangular hole in the middle; the rectangular hole is meant to be the same size as your hole (i.e. your top-level rectangular shape that I see in the picture;

so, let's call this new geometry as "Plane with hole" for convenience;

so, you will need to draw it (in OpenGL) with disabled color mask, so that it is rendered to the depth buffer, but not to the color buffer; this means that this "Plane with Hole" will be invisible, but on the other hand it will also hide the flank sides of your geometry (this is due to the way depth testing works in OpenGL).

The PlaneWithHole must be rendered aligned with the position and pose of the target; also, make sure the plane is big enough (i.e. much bigger than its hole in the middle)

Hi, thanks for the clarifications, but to me it still sounds unclear what you want to achieve; 

I'm really happy to help you if I can, but could you describe in a very detailed, step-by-step way what you want to do?

If you have any graphics explaining the concept you can also send me a private message....

A. 

Thanks for your reply, the "camera plane" I mentioned is the texture plane we get directly from the camera, a plane parellel to the camera with certain distance, this plane is from real world. and "vertical to the plane" I mean is that the mobile device is just parallel with the plane but above without bias angle.

Hi, can you explain a bit more in detail what you mean by "camera plane" (do you mean a plane parallel to the camera and located at a certain distance from the camera, or do you mean the trackable plane, or other )?

Also, I'm not sure to understand what you mean by "vertical to the plane"...

Maybe if you precise the problem better, we can probably help with some ideas on how to achieve your effect...