Hello,

AR experiences on see-through eyewear devices, such as HoloLens, are extremely sensitive to tracking jitter. For this reason, HoloLens uses a sophisticated tracking system consisting of several cameras fused with inertial sensors. Vuforia can only use a single camera which, by itself, is not sufficient to provide stable tracking and therefore always produces a combination of the tracked pose of the object and HoloLens' integrated tracking to provide stable, jitter-free AR experiences.

Minor corrections are also performed to compensate for the inevitable, smaller misalignments that happen as the user moves around the tracked object. But these corrections, as you observed, happen with a noticeable delay to avoid introducing jitter and jumps. In addition, the ADAPTIVE mode (now the DEFAULT with recent Engine SDK versions) also supports stronger object motions and automatically switches to a different mode when larger pose changes are detected. This switching is most likely introducing the sudden motion you are observing as you are moving the object.

The only way to turn this behavior off and get access to the raw camera-based poses is to use the AR_CONTROLLER (previously DYNAMIC) mode. But then, as you noted, the tracking stability is significantly reduced.

Kind regards,

Dave Downing

Vuforia Engine Technical Support

This is a follow up to my previous post of Dec. 1.  Would it be possible for you to share documentation or other details on exactly what is going on under the hood in the Adaptive and Dynamic tracking modes.   Knowing this would greatly help us debug our tracking and position popping problems.  Also, can you confirm that on the Hololens 2 the calculation of the tracked position and orientation is computed over a number of frames? From our observations it appears that there is a 3 of 4 frame lag between the actual physical position/orientation and the associated tracked position/orientation computed by Vuforia.  Thanks!

-Steve

Dave,

This is a follow up to Lennart and my posts from Sept. and Oct.    In the Hololens application we are developing we are trying to move an object with an attached image target to a desired position (with an error less than 2mm).   For the most part we are able to do this, but at times, a small change in position or orientation of the real object causes a large pop in the position and orientation of the associated graphica object.  I think this is due to use of motion hint or other compenstation algorithms that predict the future position based on the current velocity.   It also appears that when there is a large change in the position and orientation of the image target, the orientation of the associated graphical object updates immediately but the position is updated over a number of frames (which looks like it is being producted by interpolating between the original position and the new position over a number of frames).  Are there any parameter or project settings to get the postion to also update immediately?   We have tried using the Dynamic Tracking mode, but this causes the tracking quality to become very sensitive to changes in head movement (where as in Adaptive mode head movement have little to no effect on the position and orientation of the graphical object associated with the image target).  If we know the change in Hololens position and orientation in each frame can this be used to offset the Vuforia tracking errors  caused by head movements in the Dynamic mode?  Basically what we want is the raw, instantaneous position and orientation of the image target at each frame without any additional corrections due to motion hints or other compensation.  Any other suggestions you might have are most appreciated.  Thanks!

-Steve

Hello,

What you're observing are artifacts from optimizations made in our tracking algorithm to run on embedded systems.

Kind regards,

Dave Downing

Vuforia Engine Technical Support

Thanks again, Dave. I understand that there is jitter, and the resolution/accuracy is fine, but I was wondering why the jitter is different per frame in a static image? My expectation (and I might be wrong) was that the error in pose estimation should be the same/deterministic if the image does not change. Does our image target perhaps have too many feature points so that they are chosen at random leading to different results per frame or could there be another source of randomness? Thanks again in advance for all your help.

Hello @LennartM,

Quote:

We have a static testbed where we feed the same image every frame to a virtual camera and run Vuforia on that. We noticed that there is a bit of noise, up to +/- 0.2mm between different frames for the position estimate.

Pose 'jitter' will occur for all target types. The magnitude of the jitter can be impacted by a few factors including the quality of the target. Here are some tips for optimizing 2D planar targets:

https://library.vuforia.com/objects/best-practices-designing-and-developing-image-based-targets

The level of pose jitter your reporting (+/- 0.2mm) seems reasonable considering the current state of the technology.

Kind regards,

Dave Downing

Vuforia Engine Technical Support

Thank you very much Dave. We were able to experiment enough with the different modes to get a better understanding of the problem. A major source of issues with head movement on the HoloLens was the type of shader being used which was not properly compatible with depth buffer sharing, so the issue was just visual. Apologies for thinking the underlying recognition was at fault.

Now that it is a lot more stable, we can finally focus on fine-tuning. I have one question in that regard: We have a static testbed where we feed the same image every frame to a virtual camera and run Vuforia on that. We noticed that there is a bit of noise, up to +/- 0.2mm between different frames for the position estimate. This is not using AR_CONTROLLER. Is this expected for entirely still image targets (because perhaps they are never expected)? Thanks again!

Hello,

AR_CONTROLLER (DYNAMIC) is the "pure CV" mode and was implemented for just this use case.

Kind regards,

Dave Downing

Vuforia Engine Technical Support

Hello Dave,

Thank you very much for your reply. I work with shlane and we are using Vuforia version 10.5.5. When using Adaptive for our Image Target, the jitter is a lot less whereas when using Dynamic, it jitters a lot (as seen in the previous video). However, there are still issues where we observe a "pop" of the tracking which can be observed in the video attached, most visibly at 0:05, 0:09 and 0:14. Any idea what might cause this?

Also, in the last 10 seconds of the video you can see how the tracking plays catch up with the real world object more than normal. We believe this is due to Adaptive internally switching to "Static" and then upon detecting movement switching to "Dynamic". Is that correct?

Ultimately, latency is not as important to us as is accuracy, especially accuracy between frames. Is there a way to make Vuforia run pure CV without any extrapolation so that we get the position of the Image Target in the specific frame? Most of our application is geared towards static and very slow movements, but these bad "pop" frames cause lots of trouble. Thanks in advance for taking the time!

Hello @shlane,

Which version of Engine SDK is being used in the video you provided?

For more recent versions of Engine, the SDK constantly tracks the target and performs small corrections to the pose to optimize alignment, compensate for view-specific errors and wrong-sized targets. This may be perceived as "jitter".

If you do not want the above described behavior, you can detach the augmentations from the target after detection and have them tracked by HoloLens only. Then there will be jitter, but also higher misalignment the further the user moves from the initial detection position.

If you're using Engine SDK 10.9 or later, you can also try experimenting with the DYNAMIC TargetMotionHints and the AR_CONTROLLER TrackingOptimization setting.

Kind regards,

Dave Downing

Vuforia Engine Technical Support

Here's a video (especially the first 15 sec) that shows the jitter associated with objects placed on the image target when moving the head.  You can also see the tracking latency/error associated with keeping the head still and moving the image target.  I will be posting another video later today showing similar jitter in a sample application.

I also noticed that if we physically pin the position of an object, and just rotate it, we can get large position errors at the point where it is pinned.  However, when I stop rotatating the object these position errors go to zero.  It looks like vuforia uses some type of motion estimatation to predict the position of where an object will be in the future (to reduce latency and improve the apparent tracking accuracy of graphical objects).   In our application it is important that we have the most accurate position estimates possible.     Is there any way to disable the Vuforia motion compensation, estimation and/or prediction we are seeing to improve the accuracy?  

Thanks for your help.  It is most appreciated.

-Steve

https://drive.google.com/file/d/1V1WPiKrTx7Dyz6l9jSws1PaqVxP3anUx/view?usp=sharing

Hey there,

Would you be able to share in detail the similar tracking issues that you're experiencing? Another video would be very helpful of the device tracking.

Kind regards,

Patrick Scheper

Vuforia Engine Product Manager

We are also having the same issues .  That is, significant jitter associated with the position of graphical objects in the scene when the user moves their head even small amounts. Exactly as shown in the video attached to the previous posting.   If you could confirm you are aware of this issue and it is being addressed, it would be most appreciated.   Thanks.

Was there ever an update on this? I'm experiencing the same problem.

Here is a link to a video I made showing the issue: https://www.youtube.com/watch?v=sAAF9gmRjHk

Hey there,

Thanks for reporting the issue. Is there any possibility you can provide your own video of the issue on the Microsoft Hololens 2?

Do you have any reproducible steps for the issue, especially with our Hololens 2 Sample? 

Thanks and have a good weekend.

Kind regards,

Patrick Scheper

Technical Community Manager

Bump