VR only scene from scratch

Hi everyone,

I am struggling to setup a VR only scene using Vuforia prefabs. 

This is what I do

1. Added AR camera to the scene and a dded light.

2. Added some objects to the scene (just cubes and spheres)

3. Created a script and attached to an empty gameobject. in the script I use these 2 lines to switch to VR mode. 

MixedRealityController.Instance.SetMode(MixedRealityController.Mode.VIEWER_VR);

VideoBackgroundManager.Instance.SetVideoBackgroundEnabled(false);

4. When I build the scene to my phone. It starts in VR but freezes from time to time for a few seconds than resets to original position.

Am I missing something here? Any help is appreciated. Thanks in advance.