I'm trying to create a sort of GPS/compass/gyroscope AR application where I'd put various 3D objects on real-world GPS coordinates and have them appear at the right location on the screen (and the phone's camera feed along with it). The issue I'm having is that my GPS markers don't fall on the right locations, but only when viewing from some places (when viewing from other places they align perfectly). The reason I'm using Vuforia is to display the camera feed because Unity's native API (WebCamTexture) is really slow and has really bad FPS.
I already have a working way to convert GPS coordinates to Unity coordinates (from GPS to ECEF and in the end ENU), the camera rotates correctly using the gyroscope and the world orients correctly via the compass, so I think the problem is that Vuforia's camera feed does not seem to coincide with what Unity's camera displays. I'm also guessing this has something to do with FOV and the projection matrix, but I'm not really sure how to check and/or account for this. Also, if I'm correct, Vuforia has to enlarge the camera feed a bit so it fills the entire screen (without black bars in either side), so this might also affect the issue even more.
Can someone push me in the right direction and tell me what I'm doing wrong? Did anyone try to do anything similar?